Study finds AI use in trial courts results in over 6pc more cases resolved annually

A large-scale field trial on the integration of generative artificial intelligence (AI) into the country’s justice system found that the use of AI resulted in an additional 1,848 cases being resolved per year, an increase of more than six per cent over the average, it emerged on Friday.

“This increase does not appear to come at the expense of reduced quality,” observed a research paper on the rollout of a generative AI assistant in Pakistan’s trial courts, titled “Courts of Tomorrow: Evidence from a Nationwide Rollout of Generative AI”, that was published on Tuesday.

The authors of the paper, researchers Sultan Mehmood, Christoph Goessmann and Elliott Ash, said that the custom assistant — named JudgeGPT — was a chatbot based on OpenAI’s GPT-4 family of models. It was customised for the Pakistani context and beta-tested “intensively” with Federal Judicial Academy before being deployed for use by 1,559 judges serving across 118 courts.

According to the findings, it was most effectively implemented when paired with targeted training for judges on the use of the tool. The trial found that introducing the assistant alongside targeted training corresponded to an additional 1,848 cases being resolved per year, a 6.3pc increase over the mean.

“Judges who received AI access together with targeted training on the use of the tool were more likely to adopt it, use it more intensively, and continue using it over time,” the study noted. “Their attitudes toward AI also shift: they expect the tool and the targeted training to increase their productivity.”

Targeted training on the implementation of the AI tool was found to apparently shift its use towards tasks where language models were “likely to be more useful”, such as text improvement, and away from more open-ended legal queries “where responses are more costly to verify”, it added.

The trial was conducted among “roughly half of the country’s trial judges and 80pc of district courts”, with 1,559 judges randomly divided into three groups for the study.

Of these, one group was given access to JudgeGPT with targeted training on how to use the tool in judicial work, while one was given access to the tool with only “generic training” on technology and law. The control group received generic training without access to the assistant.

The outcomes were measured via a baseline survey on judge’s attitudes towards generative AI, followed by another survey roughly three months after the rollout which measured post-treatment perceptions and expected productivity gains. The records of the JudgeGPT platform were also accessed to measure the uptake and determine what kind of tasks judges assigned to the tool.

Additionally, district-level administrative court records were accessed to examine whether assigning judges to the AI tool increased local case resolution. Judicial opinions from before and after the trial were also used to assess its effects on writing quality and “whether AI altered written attitudes towards gender or religion”.

The findings showed that, although the post-treatment opinions of the “treated” judges contained more text classified as AI-generated than those of the control group, there was little evidence that this resulted in a deterioration in writing quality.

“If anything, there is a positive effect of AI on the quality assessment.” The study also noted “little evidence of systematic changes in pro-Muslim or gender bias in judicial language”.

Judges primarily used the tool for legal research and writing support, it was found, while targeted training shifted its use towards “bounded support tasks such as text improvement and summarisation, rather than full-text generation”.

“These uses are well suited to large language models and are more likely to preserve judicial agency,” the study added.

In its conclusions, it noted that access to JudgeGPT was found to increase AI use, but sustained engagement depended strongly on targeted training.

“We do not study AI as a replacement for judges,” the paper stated. “We study it as a tool that may change how judges carry out recurring parts of their work.”

It added, “For judiciaries facing persistent backlogs, AI is therefore not a panacea. But when a tool is built around relevant legal materials and paired with training that directs use toward appropriate tasks, it can become a practical tool for improving state capacity.”

In April, the National Judicial Policy Making Committee (NJPMC) formally issued national guidelines for the use of AI in judicial institutions.

The key highlights of the guidelines include a human-centric approach through which AI would assist — and not replace — judicial decision-making, ensuring judges remained the “ultimate arbiters”.

Likewise, it will promote ethical and transparent use, providing strong safeguards against bias, with emphasis on explainability and accountability.