Artificial Intelligence More Accurate Than Lawyers for Reviewing Contracts, New Study Reveals

Artificial intelligence has overtaken lawyers for the first time in a staple of the legal profession – accurately spotting risks in everyday business contracts.

In a new study, LawGeex, the leading AI contract review platform, has achieved an average 94% accuracy rate at surfacing risks in Non-Disclosure Agreements (NDAs), one of the most common legal agreements used in business. This compares to an average of 85% for experienced lawyers.

The study pitted the LawGeex AI solution against 20 US-trained top corporate lawyers with decades of experience, specifically in reviewing NDAs. The participants’ legal and contract expertise spanned experience at companies including Goldman Sachs and Cisco, and global law firms including Alston & Bird and K&L Gates.

Both the lawyers and the LawGeex AI analyzed five previously unseen contracts, containing 153 paragraphs of technical legal language (“legalese”), under controlled conditions precisely modeled on the way lawyers review and approve daily contracts. This is the first time that an AI has been tested with a typical task undertaken by lawyers on a daily basis.

The highest performing lawyer in the study achieved 94% accuracy – matching the AI – while the lowest performing lawyer achieved an average 67% accuracy. The challenge took the LawGeex AI 26 seconds to complete, compared to an average of 92 minutes for the lawyers. The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes.

Study based on contract expertise from top US academics

Leading legal academics and veteran US corporate lawyers who collaborated in the study include:

• Professor Erika J.S. Buell, Director of the Program in Law & Entrepreneurship, Duke University School of Law.
• Gillian K. Hadfield, Professor of Law and Economics at the University of Southern California.
• Bruce Mann, former senior partner at top US law firm, Morrison Foerster, who has handled more than 300 IPOs and over 200 mergers & acquisitions.

The study was overseen and administered by independent lawyer Christopher Ray. Additional consulting academics include Dr. Roland Vogl, Professor of law at Stanford Law School and Professor Yonatan Aumann, Professor in the Department of Computer Science, at Bar Ilan University.

Gillian K. Hadfield, Professor of Law and Economics at the University of Southern California said:

“This experiment may actually understate the gain from AI in the legal profession. The lawyers who reviewed these documents were fully focused on the task: it didn’t sink to the bottom of a to-do list, it didn’t get rushed through while waiting for a plane or with one eye on the clock to get out the door to pick up the kids. The margin of efficiency is likely to be even greater than the results shown here.

“This research shows technology can help solve two problems – both making contract management faster and more reliable, and freeing up resources so legajeffl departments can focus on building the quality of their human legal teams.”