‘Methyl Isocyanate of law’: SC slams AI ‘hallucinations’, quashes NCLT verdict
The bench, comprising Justices PS Narasimha and Alok Aradhe, said: “The production of fake, non-existent and hallucinatory materials and their use as precedents in law is akin to the release of methyl Isocyanate in the field of law and justice: invisible, insidious and disastrous once anyone notices it. This not only pollutes but also takes away the lifeblood of judicial determination.”
Also Read: Lawyers welcome SC rules on AI against ‘algorithmic justice’
It set aside an NCLT order on the bankruptcy of Essel Infraprojects after it found that the court relied on non-existent, fake and hallucinatory decision precedents created through AI tools.
“Courts are required to adopt a zero-tolerance mode for producing, quoting or using AI-generated precedents without verification. Citing such decisions without verification is abuse on the part of a lawyer,” the ruling said.
The panel said it was equally a “serious mistake” for a judge to rely on such fake or hallucinatory AI-generated material as precedent to support the decision.
“We have no hesitation in declaring that such a decision is not a decision in the eyes of the law, regardless of whether it has a direct or indirect impact on decision-making. Even if a modicum of spurious or hallucinatory material enters the decision-making process, such decisions should be set aside as they would violate the sanctity of the proceedings.” he said. The board said it was absolutely necessary to maintain integrity in decision-making.
“And we reiterate and declare zero tolerance for both the Bar and the Board to quote, reference or rely on such materials. It has also been clarified that our decision will have nothing to do with the proper use of artificial intelligence, but will have to do with the presentation or reliance on fraudulent or hallucinatory materials as if they were court precedent.” he said.
The bench said a mere declaration of prohibitive action is not sufficient and there must be consequential action followed by accountability.
“As far as the liability of the bar is concerned, we direct the Bar Council of India (BCI), the apex statutory body, to constitute a committee and deliberate on the issue of members of the bar submitting such fake and hallucinatory materials to the Court as if they were precedents.”
The resolution stated that the apex bar body should take this issue with utmost seriousness, consider it seriously and determine a guiding principle for preventing such incidents as well as the disciplinary action to be imposed following violation of norms.
The issue arose from an insolvency dispute involving Pooja Ramesh Singh and Jammu and Kashmir Bank Ltd and Essel Infraprojects Ltd.
The appellant challenged the NCLT Mumbai order admitting his Chapter 7 bankruptcy petition.
The apex court found that many of the “precedents” cited by the NCLT to justify its decision did not exist.
These included made-up case names and paragraphs mistakenly attributed to actual quotes. For example, the judgment referred to ICICI Bank Ltd v. Urban Infrastructure Real Estate Ltd (2019) 16 SCC 528 and Sarbjit Singh v. Union Bank of India (2022) 7 SCC 464; both of these were found to be completely non-existent quotes.
Even though the respondent, Jammu and Kashmir Bank, had filed an affidavit explaining that its lawyers had not cited these cases and that the NCLT had obtained them through its “own investigation”, the top court held that the source of the error did not mitigate the prejudice to the rule of law.
“Beyond the inevitable consequence of setting such decisions aside, what is important to our decision-making process is our determination to embrace AI technology to assist in decision-making, while at the same time asserting and declaring complete and absolute control over decision-making, with a human in the loop at every stage,” the panel said.
