Few cases ruffle Hamid Ismail after nearly two decades as a lawyer, but he was taken aback when a man he defended was sentenced with the help of an artificial intelligence tool in the Malaysian state of Sabah.

Ismail knew courts in Sabah and neighbouring Sarawak were testing the artificial intelligence tool for sentencing recommendations as part of a nationwide pilot but was uneasy that the technology was being used before lawyers, judges and the public fully understood it. There was no proper consultation on the technology’s use, and it is not contemplated in the country’s penal code, he said.

“Our Criminal Procedure Code does not provide for use of artificial intelligence in the courts ... I think it’s unconstitutional,” said Ismail, adding that the artificial intelligence-recommended sentence for his client for a minor drug possession charge was too harsh.

The courts of Sabah and Sarawak piloted software developed by Sarawak Information Systems, a state government firm, which said at the time that it had held consultations during the process and taken steps to address some of the concerns raised. World over, the use of artificial intelligence in the criminal justice system is growing quickly, from the popular DoNotPay chatbot lawyer mobile app to robot judges in Estonia adjudicating small claims, to robot mediators in Canada and artificial intelligence judges in Chinese courts.

Authorities say artificial intelligence-based systems make sentencing more consistent and can clear case backlogs quickly and cheaply, helping all parties in legal proceedings to avoid lengthy, expensive and stressful litigation. More than a third of government respondents in a global survey last year by research firm Gartner indicated that they planned to increase investments in artificial intelligence-powered systems including chatbots, facial recognition and data mining across sectors.

This month, Malaysian federal authorities aim to conclude their nationwide trial of the artificial intelligence sentencing tools, which they have said “can improve the quality of judgment”, though it is not entirely clear how they will be used in courts.

A spokesperson for Malaysia’s Chief Justice said the use of artificial intelligence in courts was “still in the trial stage”, declining further comment.

Bias, mitigating factors

Critics warn artificial intelligence risks entrenching and amplifying bias against minorities and marginalised groups, saying the technology lacks a judge’s ability to weigh up individual circumstances or adapt to changing social mores.

“In sentencing, judges do not just look at the facts of the case – they also consider mitigating factors, and use their discretion,” Ismail told the Thomson Reuters Foundation. “But artificial intelligence cannot use discretion.”

Considering aggravating and mitigating factors “requires a human mind”, said Charles Hector Fernandez, a Malaysian human rights lawyer.

“Sentences also vary with changing times and changing public opinion. We need more judges and prosecutors to handle increasing caseloads – artificial intelligence cannot replace human judges,” he added.

Seeking to address concerns that its artificial intelligence software might lead to bias in sentencing, Sarawak Information Systems said it had removed the “race” variable from the algorithm.

But while “such mitigating measures are valuable, they do not make the system perfect”, said a 2020 report on the tool from the Khazanah Research Institute, a policy think-tank.

It also noted that the company had only used a dataset of five years from 2014-’19 to train the algorithm, “which seems somewhat limited in comparison with the extensive databases used in global efforts”.

Sarawak Information Systems could not be reached for comment on whether it had since expanded its database. An analysis by Khazanah Research Institute of cases in Sabah and Sarawak showed that judges followed the artificial intelligence sentencing recommendation in a third of the cases, all of which involved rape or drug possession under the terms of the two states’ pilot.

Some of the judges reduced the suggested sentences in light of mitigating factors. Others were toughened on the basis that they would not serve as a “strong enough deterrent”.

‘Opaque algorithm’

Technology does have the potential to improve efficiency in the criminal justice system, said Simon Chesterman, a professor of law at the National University of Singapore.

But its legitimacy depends not only on the accuracy of the decisions made, but also the manner in which they are made, he added.

“Many decisions might properly be handed over to the machines,” said Chesterman, a senior director at AI Singapore, a government programme. “[But] a judge should not outsource discretion to an opaque algorithm.”

Malayasia’s Bar Council, which represents lawyers, has also voiced concern about the artificial intelligence pilot.

When courts in Kuala Lumpur, the capital, started using it in mid-2021 for sentencing in 20 types of crimes, the council said it was “not given guidelines at all, and we had no opportunity to get feedback from criminal law practitioners”.

In Sabah, Ismail appealed his client’s sentence recommendation by the artificial intelligence tool, which the judge followed. But he said many lawyers would not mount a challenge – potentially condemning their clients to overly harsh sentences.

“The artificial intelligence acts like a senior judge,” Ismail said. “Young magistrates may think it Is the best decision, and accept it without question.”

This article first appeared on Thomson Reuters Foundation News.