The race to digitize legal departments has reached a critical juncture, with a new study focused on the best Artificial Intelligence for legal teams and in-house counsel revealing surprising insights about AI performance in real-world legal settings. Following the previous benchmarks, this targeted evaluation answers the question many corporate legal teams are asking:
“Is AI ready to handle in-house legal work, or is it all marketing hype?”
Unlike the previous study, which showed legal AI outperforming lawyers in certain tasks, this benchmark study specifically examined how generative AI tools perform on tasks that in-house legal teams face daily, using real documents and queries submitted by corporate counsel from the United States, the United Kingdom, Singapore, and China. This study also compared general-purpose AI models against more specialized tools.
IN THIS ARTICLE...
Study Design: Real-World Testing for Real-World Results
The evaluation focused on information extraction tasks – a common starting point with AI adoption for in-house legal teams – and tested six different AI assistants across three categories:
- Legal AI Assistants: GC AI and Vecflow’s Oliver
- Productivity AI Assistants: Google’s NotebookLM and Microsoft’s Copilot
- General-Purpose AI Chatbots: OpenAI’s ChatGPT and DeepSeek
The methodology emphasized the practical usefulness of the best AI for legal teams, assessing accuracy and whether outputs were usable in a corporate legal setting. Each in-house counsel AI assistant was evaluated against 18 diverse information extraction tasks, with performance assessed through both an Accuracy Assessment (pass/fail) and a Qualitative Assessment measuring helpfulness, appropriate length, and feature support.
Key Findings: Best AI for Legal Teams & In-House Counsel
The study revealed several insights that challenge conventional wisdom about Artificial Intelligence for legal teams:
1. General AI Can Match Legal-Specific AI in Accuracy
Perhaps most surprisingly, general-purpose AI tools performed just as well as specialized legal AI assistants in raw accuracy. NotebookLM topped the accuracy rankings by successfully completing 14 of 18 tasks, while ChatGPT, DeepSeek, and Oliver each scored 12 correct answers. GC AI followed with 10 passes, and Microsoft Copilot lagged with only 7 passes.
This suggests that the underlying language models powering today’s AI tools have reached a level of sophistication where even general-purpose systems can understand and extract legal information with reasonable reliability.
2. Purpose-Built AI Delivers Superior Usability
While general tools matched or exceeded legal-specific AI in accuracy, the study revealed that purpose-built legal tools offer significant advantages in usability and workflow integration, important elements that ultimately determine whether the technology delivers value in practice.
When looking at “Usefulness Factors” like helpful formatting, appropriate detail level, and feature support, the rankings shifted dramatically. Oliver claimed the top position, while GC AI tied with NotebookLM for second place despite achieving fewer passes.
AI solutions for legal teams distinguished themselves through:
- Source-linked answers that allow for quick verification
- Multi-document support for complex reviews
- Structured outputs tailored for legal review processes
3. Common AI Failure Modes Identified
The benchmark identified six specific scenarios where AI tools consistently struggled. These failure modes represent crucial awareness points for legal teams implementing AI, as they highlight where human oversight remains essential.
- Open-ended questions led to incomplete answers, with tools often missing relevant clauses when the query lacked clear boundaries
- Missing information frequently resulted in hallucinated answers rather than acknowledgment of uncertainty
- Multi-document analysis proved challenging even for tools with sufficient context windows
- Leading questions with false premises caused AI to reinforce incorrect assumptions
- Technical limitations like file format issues, OCR failures, and content filters prevented proper analysis
- Contradictory information was often handled by selecting one interpretation without acknowledging the conflict
Practical Implications for Corporate Legal Teams
For in-house counsel evaluating AI technologies, the study offers several actionable insights:
Short-Term Implementation Focus
The benchmark identified specific use cases where the best AI for legal teams is already proving valuable:
- Clause and definition retrieval
- Boilerplate identification
- Policy extraction
- Contract triage and routing
- Obligation and deadline tracking
- Risk and metadata tagging
These represent low-risk, high-return starting points for legal departments seeking to implement AI technologies.
Looking Beyond Current Accuracy
As AI models continue to evolve rapidly, accuracy differences between tools are likely to narrow quickly. Forward-thinking legal departments should evaluate AI platforms based on:
- Intuitive interfaces that minimize training requirements
- Integration with existing document management systems and email
- Strong data security features that meet corporate compliance requirements
- Responsive vendor support that understands legal workflows
- Scalability to handle increasing document volumes
Human Oversight of Artificial Intelligence for Legal Teams
Despite impressive capabilities, the study confirmed that human judgment remains critical in the legal AI implementation process. Even with the best Artificial Intelligence for legal teams, real people are still needed to:
- Frame queries clearly to avoid incomplete answers
- Interpret ambiguous results
- Verify outputs, especially for complex or high-stakes matters
- Identify and resolve contradictions that AI might miss
Breaking Down Barriers: The Human Element
While the benchmark results highlight the technology’s capability, many in-house legal teams still face significant hurdles to adoption. Mariette Clardy, an Assistant General Counsel at a Financial Services Firm who also runs hands-on workshop series for in-house counsel, sees this firsthand in her work simplifying AI for legal teams.
“For in-house attorneys, many of them are curious and want to learn, but what gets in the way is not having a path or resources internally to get them there. But even that is changing as attorneys who do have experience like myself are supporting others even outside of our own enterprises.”
This informal knowledge-sharing network points to a growing trend where AI-savvy legal professionals act as bridges, helping colleagues overcome the initial learning curve. Such peer support may prove just as valuable as the technical capabilities of the tools themselves, especially for legal departments without dedicated innovation resources.
Clardy’s comment aligns with the study’s findings on usability – tools that can be quickly understood and applied by legal professionals with varying technical backgrounds will likely see faster adoption and deliver greater value, particularly when paired with guidance from experienced peers.
This observation is echoed by Nicole Black, Legal-tech Journalist and Principal Legal Insight Strategist, who has studied adoption trends closely.
“I’ve found in-house counsel to be really interesting when it comes to AI. I think they are one of the best use cases for it, but the data shows they’re approaching AI more cautiously than other lawyers. I can see how the lack of resources could impact moving forward with it.”
This resource gap creates a paradox: the legal teams who might benefit most from AI’s efficiency gains are often the most constrained in their ability to implement it.
What This Means for Legal Tech Strategy
This benchmark study suggests we’re entering a new phase in AI innovation, one where the key differentiator of artificial intelligence for in-house lawyers and legal departments isn’t raw text generation capability but rather how well tools integrate into legal workflows and augment human expertise.
As accuracy becomes a baseline expectation across Artificial Intelligence for legal teams, the real value for corporate legal departments will come from platforms that streamline legal work through intuitive interfaces, seamless integration with existing systems, and features specifically designed for legal review processes.
The findings also highlight the importance of a clear-eyed evaluation of AI limitations. The identified failure modes aren’t just academic concerns, they represent real risks that must be mitigated through proper implementation strategies and human oversight.
Looking Ahead: Transparency & Evaluation
The benchmark authors (Anna Guo and Arthur Souza Rodrigues) acknowledge several limitations in their study, including its narrow task scope, English-only evaluation, and snapshot-in-time nature given the rapid pace of AI development. They plan to expand their evaluation to cover other legal functions and include new AI assistants in future reports.
For corporate legal departments, the message is clear: the best Artificial Intelligence for legal teams has reached a level of capability that makes it valuable for many routine information extraction tasks, but the choice of platform should consider not just accuracy but also usability features, integration capabilities, and support for legal-specific workflows.
As the legal technology landscape continues to evolve, ongoing evaluation and transparency from vendors will be critical in helping legal departments make informed decisions about which AI tools deliver value in corporate legal settings.
Ricci Masero is an accomplished Chartered Marketer and content creator with over 25 years of experience in the field. Alongside marketing the award-winning blended learning solutions from Intellek, Ricci has been writing about technology and innovation for many years – his thought leadership has featured in Forbes, Entrepreneur, Alassian, and AI Journal – as an author, he is a regular contributor to eLearning Industry, Training Journal, and many other industry publications. He is also co-host of the L&D Insights segment in Intellek’s client webinars.
