🔍 Read the full analysis: Ironclad’s Fine Print Meets OpenAI’s Plan To Train Agents In Software on ThorstenMeyerAI.com
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI described training and testing a frontier model inside Ironclad’s contract-management software, using 11 legal, commercial and procurement tasks. GPT-6 Astra met an average 55% of task criteria, while OpenAI’s time estimates were simulated rather than measured customer savings. The work also signals OpenAI’s effort to enlist software companies as training partners.
OpenAI said it trained and evaluated GPT-6 Astra on workflows inside Ironclad’s contract-management software, reporting an average 55% of rubric criteria met across 11 tasks. The October 6 post also invites a small number of other software companies to work with OpenAI on agent training, but the results do not establish that the model is ready to run contract workflows without human review.
The tasks were selected by Ironclad staff and OpenAI employees who use the product and covered legal, commercial and procurement work. Examples included setting up nondisclosure agreements, creating procurement approval processes and revising a reusable clause based on a requester’s jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes on each task.
Each task was scored against a rubric of 8 to 50 criteria, depending on its complexity. OpenAI reported that GPT-6 Astra met an average 55.0% of those criteria, compared with 41.6% for GPT-5.6 Sol at its high setting. An internal OpenAI model used during Astra’s development scored 63.7%. On one showcased task, Astra met about 94% of the criteria; that single result does not represent its average performance.
OpenAI said Ironclad supplied hosted copies of its product for model practice. It also said it created synthetic tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. OpenAI stated that it did not use its customers’ data, its internal contracts or non-public Ironclad customer data for the work.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Contract Rules Matter to Agents
The results show both the potential and the limits of training agents in the software where professional work happens. A model that can follow multi-step instructions within a product could reduce repetitive work. But a 55% average rubric score is not a 55% task-completion rate, and it does not indicate that a workflow is safe or useful in production.
In contracting and procurement, missed requirements can defeat the purpose of a process. For example, an approval workflow may need Finance sign-off above a spending threshold, Security review for certain requests and Legal review for nonstandard terms. If an agent misses one rule, the result could bypass a required control. OpenAI’s post acknowledges that losing track of a business rule limits what a software company can confidently delegate and says human oversight remains important.
OpenAI estimated 19.2 minutes per Astra attempt, against 37 minutes for GPT-5.6 Sol. Those figures may sound like a time saving, but they are simulated estimates based on assumed processing and generation speeds—not measured customer results. Combined with the average rubric score, they do not show that Astra can complete the work faster than a person while meeting every requirement.
Products Worth Considering
Anker SOLIX S2000 Portable Power Station, 2,010Wh, 1,500W
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
As an affiliate, we earn on qualifying purchases.
How the Ironclad Test Was Set Up
OpenAI framed the project as an effort to improve models’ ability to understand business rules, perform multi-step work in specialized software and check the finished work against its requirements. The Ironclad product served as a hosted test environment; the 11 tasks were selected for the research rather than drawn from a broad measurement of customer activity.
The choice of contract-management software matters because the work depends on more than navigating screens. It can involve linked approvals, jurisdiction-specific language and controls that must be followed throughout a process. Ironclad’s chief technology officer, Sunita Verma, emphasized the need for agents to preserve “the controls teams rely on.” OpenAI also presented the project as evidence that a full contracting platform remains important, even as agents take on more tasks within it.
The final part of the post broadens the effort beyond Ironclad. OpenAI said it is inviting a small number of software companies to bring tasks that current agents cannot reliably complete, subject-matter experts, secure test environments and data that can be used safely for research. That proposal makes software vendors potential partners in training agents for specialized work.
What the Test Does Not Show
The post describes results on 11 research tasks; it does not establish how the model would perform across the wider range of Ironclad customer workflows. The average criteria score also does not identify, in the supplied material, which individual requirements were missed across each task. That matters where one missed approval or clause requirement can make a result unusable.
OpenAI’s time figures are estimates, not observed completion times or customer productivity data. The source material also does not specify a deployment timetable, the extent of human review in any future customer use, or how responsibility would be assigned if an agent missed a required control. No customer rollout or production-level accuracy claim is confirmed here.
The proposed work with other software companies remains an invitation, not evidence that additional partnerships have been completed. The material does not name other partners or provide terms for participation, data access or commercial arrangements.
OpenAI’s Proposed Partner Program
OpenAI says it plans to work with a small number of software companies on difficult tasks agents cannot yet complete reliably. Prospective partners are expected to supply concrete examples of failure, people with deep knowledge of the work, secure environments for testing and data suitable for research.
Further reporting would need to show which companies participate, what safeguards govern their data and how models perform across a wider range of tasks. For buyers, the practical next step is to ask vendors for task-level evaluation results, the specific controls tested and a clear account of where human review is required. Until such evidence is available, the Ironclad results describe a research effort—not a verified customer time saving or a basis for unattended contract work.
Key Questions
What did OpenAI and Ironclad test?
They tested a model on 11 legal, commercial and procurement tasks within hosted copies of Ironclad’s contract-management software.
What does the 55% result mean?
It is the average share of rubric criteria GPT-6 Astra met across the tasks. It is not the percentage of tasks completed and does not mean the workflows were safe to use without review.
Did the test show customers would save time?
No measured customer time savings were reported. OpenAI described the time figures—19.2 minutes for Astra and 37 minutes for GPT-5.6 Sol—as simulated estimates based on assumed processing and generation speeds.
What data did OpenAI say it used?
OpenAI said it built synthetic tasks from publicly filed contracts in the SEC’s EDGAR database and filtered out personal information. It said it did not use OpenAI customer data, internal contracts or non-public Ironclad customer data.
Is OpenAI expanding this work?
OpenAI said it is inviting a small number of software companies to participate in research on difficult agent tasks. The source material does not name other partners or confirm additional collaborations.
Source: ThorstenMeyerAI.com
