AI hallucinations lead to judicial incidents! Lawyer who abused ChatGPT faces severe penalties

The New Mexico Supreme Court of the US recently ruled on a highly cautionary AI judicial case. Senior lawyer Stephen Aarons, with 40 years of criminal defense experience, was found guilty of contempt of court for submitting an appeal brief containing a large amount of false content and was punished by the judiciary. This case originated from an appeal of a murder case. The defendant was sentenced for life in prison and the defendant's family entrusted Aarons with the appeal for a retrial.

Aarons used the o3 version of the OpenAI's ChatGPT to help sort out case documents and speed up his work. But he did not pay attention to the built-in weaknesses of common AI tools. He submitted his appeal paper in August 2025. This neatly arranged document contained a lot of fake information made by AI. The brief included multiple witnesses who did not exist at all, along with forged police testimony and witness statements, and made false descriptions of past judicial precedents. The court quickly noticed the anomaly in the document.

The core issue of this incident lies in the overlooked technical vulnerability of the AI hallucination technology. To produce smooth and complete text content, the AI model will autonomously fabricate non-existent facts, characters, and materials. This is a common problem of current large language models and is not an accidental failure. Aarons' operation process was very rough. He first used an AI transcription tool to convert the audio of the court trial into text, and then imported all the case facts, disputed points, and investigation materials into ChatGPT to let the AI generate the court summary and complaint content.

He personally believed that the new AI model had extremely high accuracy and was suitable for professional fields such as law and medicine. The output content was absolutely reliable and no manual verification was conducted throughout the process. However, in the end, the content generated by the AI was full of flaws. It not only fabricated police witnesses but also fabricated the testimony of multiple ordinary people. Unlike other AI violations, this complaint did not refer to false precedents but distorted the details of the real case, causing more damage to the judicial process. During the case hearing, the judge team severely criticized Aarons' actions.

AI hallucinations are a widely known technical issue today. Still, an experienced senior lawyer chose to ignore this risk. Aarons thought he would get lucky, and this led to multiple counts of professional misconduct. He failed to carefully check every word of the complaint before filing it. He also did not tell his client and the client's family that he used ChatGPT to work on the case. When questioned, he said it was just an "honest mistake", but the court did not believe him. The court threw out all complaint papers submitted by Aarons. New public defenders were arranged for the people in the case, and the trial was moved to the 2026–2027 court term.

At the end of the incident, Aarons proposed that the court introduce new regulations, requiring legal documents to include proof of AI usage compliance, in an attempt to avoid similar problems. However, the judge directly rejected this proposal, emphasizing that the core issue is not that AI tools have no regulations, but that practitioners have abandoned the core responsibility of manual review.

This case serves as a warning for all AI users. Currently, the technology of large models is not fully mature yet, and problems such as hallucinations, distortions and fabricated content can not be completely eradicated. There is no absolutely "zero-error" AI tool. In rigorous professional fields such as law and medicine, the requirements for the authenticity and accuracy of information are extremely high, and the tolerance rate is almost zero. Over-reliance on AI, blindly believing in the accuracy of AI technology, and giving up the manual review process will not only lead to work errors but also violate professional norms and industry red lines, ultimately incurring practical compliance and legal costs.