An AI Tool for Review Across Hundreds of Thousands of Legal Documents

Summary:

By developing a refined, repeatable process that tailored AI prompts and incorporated expert feedback, a company achieved highly accurate results, with an average recall of over 96% and precision of 71%. This AI-driven approach significantly reduced review time and costs,, while maintaining exceptional accuracy.

Client: 

JND eDiscovery is a legal service provider

Problem Statement: 

JND was assigned the responsibility of reviewing millions of documents in a complex class-action lawsuit involving several corporate entities. They recognized that leveraging generative AI would greatly help their corporate client in efficiently fulfilling their production requirements.

Results: 

• Achieved an average recall of over 96% and precision of 71% across multiple analyses.
• Delivered remarkably accurate results that greatly surpassed traditional review.
• Reduced review time by hundreds of hours and cut costs.

AI Solution Overview:

JND eDiscovery participated in the limited availability program for Relativity’s aiR for Review, which leverages generative AI to mimic human reviewers by identifying and summarizing relevant documents based on user-provided criteria. To enhance this process, JND created a streamlined, repeatable method for refining prompt criteria and ensuring both accuracy and scalability of aiR. Here’s their approach:

1. Data Refinement: The team reduces the document set using search terms, date filters, and other analytical tools, customized for each case.
2. Initial Prompts: JND collaborates with their client to create initial prompt criteria. aiR for Review then uses these inputs to analyze small, representative samples — including stratified samples covering key categories and diverse samples representing the entire data set.
3. Feedback Loop:Attorneys or subject matter experts review the results, and JND refines the prompt inputs based on their feedback.
4. Quality Check: JND runs aiR for Review on a random, statistically valid sample of approximately 400 documents, calculating recall and precision metrics to evaluate the AI’s performance.
5. Full Review Execution: Once the analysis meets expectations, aiR for Review is applied to the entire document set.

For this case, JND tailored the prompt criteria in collaboration with their client, adjusting and validating the inputs to meet production requirements for three specific companies involved. This precise approach ensured aiR for Review delivered the best possible analysis.

References: 

1. JND Saves Corporate Clients Hundreds of Review Hours and Cuts Costs by Nearly 60% with Relativity aiR for Review. https://www.relativity.com/resources/customers/jnd-air-for-review/

Industry: Legal Services

Vendor: Relativity

Client: JND eDiscovery