A Vals AI and NextLadder Ventures evaluation of 23 AI models found weaker results on local legal referrals than on explanations of law, testing how well systems might serve as an initial guide for people with civil legal problems.
On rubric scoring, models performed better when asked to explain legal concepts, meeting 78% to 87% of criteria. When a task required identifying a specific service, such as a local legal-aid group or court self-help center, scores ranged from 30% to 50%. The benchmark measures whether systems met expert-defined standards, not whether anyone secured counsel or resolved a dispute.
National legal directories appeared in at least 70% of final responses from 21 systems. Vals AI's experts cautioned that a general directory may leave users to determine which listed program serves their area, making it less useful than a location-specific referral.
Qwen 3.8 Max led overall at 76.37%, followed by Claude Sonnet 5.5 at 74.94% and Wonder Full at 74.68%. Vals AI said the gaps among the top three were within the benchmark's margin of error. It set 80% of rubric criteria as its threshold for a dependable first contact; all 23 systems scored below it.
The published evaluation used 252 scenarios from a pool of 504 developed with legal-aid experts. They involved state law in California, Louisiana, Maryland, New York, Texas and Virginia; the other half of the pool was reserved for private validation. In simulated exchanges, models could search the web and ask follow-up questions. GPT-5 played the person seeking help, while the automated evaluator Decider v1.2 scored each exchange against expert-written rubrics.
The Legal Services Corporation's (LSC) 2022 study documented unmet need: for 92% of substantial civil legal problems reported by low-income Americans, help was absent or inadequate. The study also found that resource shortages left legal aid organizations unable to meet demand, with 49% of requests turned away. Among people who did not seek legal help for one or more problems, 46% cited cost concerns as a reason.
Vals AI and NextLadder Ventures said a fuller report and a brief for legal aid practitioners are in preparation.
