By Hon. Bernadette T. Clark
With each day that passes, it becomes increasingly clear that Artificial Intelligence (AI) is no longer a distant concept of the future; it is part of our daily lives. Likewise, AI also is more frequently becoming an issue within our judicial system.
As a Judge, I cannot ignore the profound ethical challenges AI presents. Nor can I ignore the potential that AI poses to influence and reconfigure legal proceedings. This article focuses on AI from a Judge’s perspective, including its impact, benefits, and risks, as well as the role of human judgment in the judicial system.
AI—particularly, generative AI—is increasingly being used by attorneys, self-represented litigants, and members of the judiciary, to draft legal documents including motions, decisions, and court orders. Although AI can generate briefs, memoranda of law, and other legal papers with staggering efficiency, it occasionally “hallucinates,” meaning AI platforms sometimes literally make up laws, statutes, or case citations that do not exist.
Filing papers that contain AI hallucinations can be a very costly mistake for lawyers and self-represented litigants. Numerous lawyers have been sanctioned by Judges—sometimes severely—for submitting court filings that contain cases or other citations hallucinated by AI. AI hallucinations are dangerous because the information they produce looks plausible on its face but in fact does not exist. When attorneys and litigants cite non-existent legal authorities created by AI, they mislead and misinform courts. Attorneys and litigants who rely upon AI-generated legal research without verifying that the resulting citations are accurate risk misinterpreting the law and suffering devastating professional and ethical consequences.
Effective June 1, 2026, the New York State Unified Court System issued Part 161 regulating the use of AI by attorneys in the New York State Courts. Under this rule, the use of AI is permitted in preparing documents that are submitted to New York State Courts. Although many states require lawyers to disclose the fact that they used generative AI in creating their submissions, New York does not require such a disclosure.
Part 161 defines AI as:
“a machine–based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments, and that uses machine- and human-based inputs to perceive real and virtual environments, abstract such perceptions into models through analysis in an automated manner, and use model inference to formulate options for information or action.”
Under the model rule, Section 161, Appendix A, attorneys and litigants are required to carefully review any paper filed with a Court and independently ensure that it contains no fabricated or fictitious cases, statutes or other relevant materials. According to this model rule, attorneys and self-represented litigants are held to the exact same standards, and no distinction will be made if this requirement is not met. If this requirement is violated, the violating attorney or party may be subject to sanctions or other actions as the Judge finds appropriate.
The use of AI in court papers is increasing, especially with self-represented individuals. While these submissions are acceptable, Courts are growing increasingly concerned that self-represented litigants may have difficulty distinguishing between real and hallucinated cases and statutes. This is especially worrying because there have been several reported cases where even attorneys have overlooked non-existent citations in their submissions because, on their face, the AI-generated “hallucinated” cases look very real.
AI is an amazing tool and using it undoubtedly saves time. However, the user must beware! To avoid problems, be sure to check and double-check any Court submissions thoroughly and carefully. As the timeless adage goes, trust but verify…especially when using AI.
