TL;DR
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Lexassay, a legal AI test run by Thorsten Meyer AI, gave 42 model configurations the same 15 questions across German, U.S. and EU law. Claude Opus 5.5 scored highest at 79 out of 100, while a model marketed for legal workflows scored 8; the test operator disclosed that Claude agents graded answers and that a Claude model ranked first.
Thorsten Meyer AI published results on November 4 from Lexassay, its test of 42 AI setups answering the same 15 legal questions. Claude Opus 5.5 led with 79 out of 100, while GLM-5.3 FlashX Legal, described by its maker Z.ai as optimized for legal workflows, scored 8; the test operator disclosed that Claude agents graded the answers and a Claude model ranked first.
The test covered German, U.S. and EU law, including deadlines, recent legal changes, court decisions and questions built around false assumptions. Models answered without internet access or tools. Lexassay says it grades each answer against a prepared answer using weighted criteria, and checks citations against official databases. The comparison included different effort settings, so some models appear more than once among the 42 setups.
Among the reported scores, GPT-6 Astra scored 72 and GPT-6.1 Sol scored 71, following Claude Opus 5.5. Mistral Large 4 scored 10, and GLM-5.3 FlashX Legal scored 8. Lexassay says GLM-5.3 FlashX Legal fully solved none of the 15 tasks and left two unanswered. These are scores from the test’s particular questions and grading rules, not a measure of every legal task a system might perform.
The test operator said results were graded blind by several Claude agents and acknowledged that this creates a conflict of interest because a Claude model ranked first. Lexassay’s method page, according to the source material, publishes the rules, public answers, prepared answers and citation checks for review. The source describes this as a spot check, with one answer per model and task, and cautions that small score differences may be noise.
How 42 AI Setups Scored On The Same 15 Legal Questions
Lexassay compared AI setups on questions spanning German, U.S. and EU law. The results show how these systems performed under one specific test, not on every legal task.
Reported scores
The test’s top results were closely grouped, while two other systems scored much lower.
What the test covered
Models answered without internet access or tools; the questions included deadlines, legal changes, court decisions and false assumptions.
Three legal systems
Questions covered German, U.S. and EU law.
Prepared criteria
Answers were assessed against weighted criteria, with citations checked against official databases.
Published and hidden
Ten questions were published and five were hidden; only scores are provided for hidden tasks.
Why citations and deadlines matter
Lexassay reported citation problems across its published runs, and tested a deadline scenario where a regional holiday changed the appeal date.
- 2,644 citations confirmed as real
- 804 identified as invented or not supporting the claim
- Some models gave incorrect appeal or filing deadlines
- Check cited laws and court decisions at their source
- Do not rely on a chatbot alone for a legal deadline
- Consult a lawyer when deadlines, money or an authority are involved
Limits to keep in view
The scores reflect a spot check under defined conditions, with one answer per model and task.
Citation Errors Shape the Scores
The results draw attention to a practical weakness for people using chatbots to research legal questions: a plausible answer can cite a law or ruling incorrectly. Across all published runs, Lexassay reports confirming 2,644 citations as real and identifying 804 as invented or not supporting the claim made. Those totals are the test operator’s findings; they do not establish how often legal AI systems make citation errors in everyday use.
Some errors can have concrete consequences. In one test scenario, two parties receive civil judgments on the same day in Munich and Berlin and ask when to appeal. The appeal deadline differs by a day because Corpus Christi is a public holiday in Bavaria but not Berlin. The source says most strong models answered correctly, while one gave the Berlin date a day late and another put the deadline for written grounds a month late. In real proceedings, a late appeal can be rejected, making deadline calculations a high-stakes task.
Lexassay’s findings support treating chatbot answers as a starting point that needs verification. They do not show that every model will fail on every deadline or citation, and the test did not allow web search or other tools that might change an answer. The operator advises users to check cited sources and consult a lawyer where deadlines, money or an authority are involved.
Products Worth Considering
DJI Osmo Mobile 8 Gimbal with Mic Mini Transmitter, Infinity Black
Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds, Black
Amazon Kindle Paperwhite Signature Edition, 32GB (2024), Metallic Black
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay asked each setup the same 15 legal questions: ten published and five hidden. The questions span German, U.S. and EU law. The hidden questions are not published in full, according to the source, so future models cannot simply learn their answers from the web; only scores are provided for those tasks.
Answers were assessed against prepared answers containing three to six weighted criteria. Citations were checked against official databases and classified as real, misattributed, invented or not findable. The method also deducts points for certain problems: each misattributed or invented citation costs a quarter of a task’s points, and an answer without a legal-advice disclaimer or referral to a lawyer loses 5 percent, the source says.
The test is distinct from benchmarks in which systems can work from supplied documents. The source notes that Vals AI’s legal agent benchmark ranks Mistral Large 4 eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. Those rankings reflect a different benchmark setup and should not be read as a direct contradiction of Lexassay’s results.
““one answer per model and task, so a few points of difference are noise.””
— Thorsten Meyer AI, the Lexassay operator
Limits of This Legal AI Test
The results do not establish how the systems would perform with web access, legal research tools or supplied case documents, because Lexassay tested answers without internet access or tools. Nor does one response to each question show how consistent a model would be across repeated attempts. The source calls the exercise a spot check and says small score differences may reflect noise.
The grading relationship also matters when weighing the ranking: the test operator disclosed that Claude agents graded blind and that a Claude model came first. Publishing the method and checks makes the process more reviewable, but the supplied material does not include an independent audit of the grades. The reported citation totals and scores therefore remain findings from Lexassay’s own test.
It is also unclear from the source material how these particular scores translate into performance on real legal work, or how the systems compare when used with lawyers, current databases or specialized workflows. The test measures a defined set of questions under defined conditions.
Check Sources Before Acting
The source points readers to Lexassay’s method page, where it says the public answers, prepared answers and citation checks can be examined. Future readers can use those materials to assess how the scoring criteria were applied; the five hidden tasks are represented by scores rather than public answers.
For users facing a legal deadline, contract issue or communication from an authority, the test operator advises checking every cited law and court decision in its source and consulting a lawyer. No deadline should be taken from a chatbot alone, the source says. The supplied material does not announce a date for a follow-up test or a new evaluation.
Key Questions
What did the Lexassay test measure?
It compared 42 AI setups answering 15 questions about German, U.S. and EU law without internet access or tools. Lexassay graded answers against prepared criteria and checked citations against official databases.
Which AI setup scored highest?
Claude Opus 5.5 scored 79 out of 100, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The operator disclosed that Claude agents graded the answers and that a Claude model ranked first.
How did the model marketed for legal work perform?
GLM-5.3 FlashX Legal scored 8 out of 100. Lexassay says it fully solved none of the 15 tasks and provided no answer on two.
Does this test show that AI can safely calculate legal deadlines?
No. It reports performance on a limited set of questions under specific test conditions. The source describes deadline errors that could matter in court and advises users not to rely on a chatbot alone for a legal deadline.
Why do these results differ from another legal AI ranking?
The source says Vals AI’s legal agent benchmark supplies documents to models, while Lexassay tests models without internet access or tools. Different tasks and conditions can produce different rankings.
Source: Thorsten Meyer AI
