TL;DR
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Harvey’s Legal Agent Benchmark, placing seventh among 81 models on Vals AI’s leaderboard. Its score was 92.5% on individual criteria, but a task fails if even one required criterion is missed. Z.ai’s claim that it leads open-source models is not fully assessable from the material provided because the comparison set is not named and published weights for the legal model were not found.
Z.ai’s GLM-5.3-FlashX-Legal passed every required criterion on 19.17% of tasks in Harvey’s Legal Agent Benchmark, ranking seventh among 81 models listed by Vals AI. The score reflects an all-or-nothing grading rule: a task counts as complete only when every criterion is met, so the result does not mean the model met only 19.17% of all rubric requirements.
Vals AI’s archived leaderboard lists the legal model at 19.17% and a test cost of $0.15. The same benchmark run shows the model meeting 92.5% of individual criteria. Z.ai says Vals independently evaluated the results. The leaderboard’s task pass rate and criterion-level score describe different things: each task can include many requirements, but a single missed requirement disqualifies the entire task from the all-pass score.
Z.ai’s developer documentation describes GLM-5.3-FlashX-Legal as a model for document understanding, legal reasoning and task execution, including research, contract review and drafting initial drafts. The company says it used legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of task environments modeled on legal workflows. It recommends connecting the model to services for laws, cases and company information. Z.ai lists API pricing at $0.37 per million input tokens and $1.25 per million output tokens.
The model is built on GLM-5.3-Flash, according to Z.ai. Its AutoClaw blog announced a beta available by application on October 13, 2026; Vals lists the release date as October 12. Z.ai says the legal model improved by 12.5 percentage points over the base model. That comparison matches the figures in the same leaderboard table, while the company’s separate claim of state-of-the-art performance among open-source models is harder to evaluate from the available material.
A 19.17% full-task pass rate still ranks seventh
Z.ai’s GLM-5.3-FlashX-Legal placed seventh among 81 models on Harvey’s Legal Agent Benchmark. Its score counts only tasks where every required criterion passes.
Why the scores differ
A task can contain many requirements, and one missed item makes the whole task a non-pass.
- Every required criterion must pass.
- Measures complete deliverables under the rubric.
- Measures individual requirements passed.
- Does not show how often work is ready to use.
What the benchmark tests
Harvey’s benchmark asks agents to produce legal work products from instructions and client matter materials.
The task includes an instruction and relevant materials.
The agent must address rubric items such as citations, deadlines and formatting.
A single missed requirement means the task does not count as complete.
What firms still need to check
A leaderboard result does not establish reliability on real client matters or the amount of lawyer review needed.
- Verify citations against primary sources.
- Measure how often drafts need correction.
- Ask how client data is processed and protected.
- Assess performance on representative work.
Why All-Pass Scores Matter to Firms
The leaderboard result places GLM-5.3-FlashX-Legal near the top of this evaluation, while also showing how demanding legal work products are under its scoring rule. A draft can contain substantial correct analysis and still fail a task if it misses one required citation, deadline, risk or formatting instruction. The 19.17% task score is therefore a measure of complete deliverables under the benchmark rubric, not a general measure of legal knowledge or the percentage of every answer that is correct.
For law firms and legal departments, that distinction matters when evaluating whether a system can assist with work that must be checked before it reaches a client or decision-maker. The benchmark’s high 92.5% criterion-level score suggests the model often satisfies individual requirements, but does not establish that its outputs are dependable as finished work products. The practical questions remain how often lawyers must repair drafts, whether citations can be verified, and how client information is handled when the model is deployed.
Products Worth Considering
Amazon Kindle Scribe (64GB) with Premium Pen, Metallic Jade
DJI Osmo Mobile 7P Phone Gimbal with Mic Mini Transmitter, Infinity Black
Bose SoundLink Flex Bluetooth Speaker (2nd Gen), Waterproof, Dustproof, USB‑C, Up to 12 hr Battery, Black
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark (LAB) as open source on May 6, 2026. Its public set contains 1,250 tasks across 24 legal practice areas. Each task pairs an instruction with a client matter containing relevant materials and asks an agent to produce a work product. More than 75,000 expert-written rubric criteria cover matters such as facts, conclusions, citations, recommendations, deadlines, dollar amounts and formatting.
Harvey calls its scoring method all-pass grading: every criterion must pass for the overall task to count as complete. One of Harvey’s examples, a change-of-control assignment, has 57 criteria. For the leaderboard, Vals AI says it uses a private holdout set of 120 tasks that mirrors the public set’s mix, with internet access disabled. Two language-model judges grade each submission, and the task pass rates they produce are averaged.
Vals AI’s leaderboard provides a comparison across 81 models under that evaluation setup. Z.ai’s documentation says its model achieved an all-criteria rate of 19.2%, up 12.5 percentage points from the base model, and described that as state-of-the-art among open-source models. The archived leaderboard gives the more precise figure of 19.17% and shows the model in seventh place.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its Legal Agent Benchmark
Open Questions on Model Reliability
The benchmark score does not establish how the model performs across real client matters or how much lawyer review its drafts require. The source material describes a small spot check of four tasks in which the legal model cited rulings that could not be found. That check is not a benchmark and cannot establish a general error rate. In the same spot check, two Claude models reportedly cited a real ruling for the opposite of what it held, while the OpenAI and Claude models cited no rulings that could not be found.
Z.ai’s statement that the model has state-of-the-art performance among open-source models is also difficult to verify from the information available: the company does not name its comparison set, and published weights for the legal model were not found. The model’s documentation presents its outputs as drafts for legal professionals. The reviewed Z.ai pages did not include a “not legal advice” notice, but the absence of such wording on those pages does not settle how the system should be used or what safeguards apply in a particular deployment.
Details about client-data processing, applicable rules and the model’s performance on specific legal tasks are not established by the leaderboard result. Firms would need deployment-specific information before deciding whether the system fits their confidentiality, verification and procurement requirements.
What Firms Should Check Next
Z.ai’s AutoClaw beta is available by application, according to its October 13 announcement. The company recommends using the model as an agent connected to services that supply legal and company data. The available source material does not provide a timeline for broader access or describe additional benchmark releases.
For prospective users, the next step is to evaluate the model on representative work under the firm’s own review process. That includes checking citations against primary sources, measuring how often drafts need correction, and asking Z.ai how client data is processed and protected. Further public details about the model’s comparison set, weights and performance outside the benchmark would help clarify how far the leaderboard ranking generalizes.
Key Questions
What does the 19.17% score mean?
It is the share of benchmark tasks on which the model passed every required rubric criterion. Under Harvey’s all-pass rule, missing even one criterion makes the whole task a non-pass.
How did the model rank?
Vals AI’s archived leaderboard lists GLM-5.3-FlashX-Legal seventh among 81 models, with a 19.17% all-criteria pass rate.
Why is the criterion-level score higher?
The same run shows the model meeting 92.5% of individual criteria. A task may contain many criteria, and it counts as complete only if all of them pass, so the task rate is lower than the criterion rate.
Does the result prove the model is the best open-source legal AI?
No. Z.ai makes a state-of-the-art claim for open-source models in this evaluation, but the material does not name the comparison set, and published weights for the legal model were not found. The leaderboard confirms its seventh-place ranking across 81 listed models, not the full scope of Z.ai’s claim.
Can the model’s drafts be used without lawyer review?
The benchmark does not establish that. Z.ai describes the outputs as drafts for legal professionals, and the reported spot check found citations that could not be located. Firms would need to verify citations and assess the model on their own work before relying on its outputs.
Source: Thorsten Meyer AI
