🔍 Read the full analysis: GPT-6 Astra’s Second Opinion On Claude’s Model Edit Caught Bugs on ThorstenMeyerAI.com
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI’s GPT-6 Astra, running in Codex, reviewed an experiment in which a Claude Opus 5.5 session edited a local model called Ornith 1.5. The review found reproducible problems in the benchmark rig and weight conversion, and challenged several claims about the results. The 10 published tasks showed more answered questions after editing, but the legal score remained zero.
An OpenAI GPT-6 Astra review found reproducible faults in the benchmark tools and weight conversion used in a Claude Opus 5.5 experiment that edited the local model Ornith 1.5. The review led to code fixes and narrower claims about the results; on the 10 published tasks, the edited model answered all 10 rather than 4, while its legal score remained zero, according to ThorstenMeyerAI.com.
The Claude session ran the experiment, including the model edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra, operating in Codex, was assigned to review the implementation rather than conduct a competing model-improvement experiment. The source says the reviewer did not access hidden benchmark prompts, topics, answers or reference solutions, and did not change the main test rig.
In its first review on Nov. 4, GPT-6 Astra reported four problems. Resuming a run could reuse results from a different model or sampling setup; an unknown task ID could finish as a successful run with no evaluations; a comma in the sampling syntax conflicted with the command-line format; and converting weights back to a compact format altered values even when the requested edit strength was zero.
The experiment team says it fixed those issues: it changed the sampling syntax, made unknown task IDs fail, adjusted how edited tensors are stored, and added tests that fail against the old code. The source reports that in a second review round, all 12 tests written by the author passed, while four of GPT-6 Astra’s six provenance checks failed. That result describes those specific checks, not a general error rate for the project.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why the Review Changed the Results
The review matters because a benchmark can report plausible-looking results even when its run controls are unreliable. Reusing answers across different settings could make an old result appear to belong to a new run, while an unknown task ID could make a test look complete despite evaluating nothing. Those faults bear directly on whether reported comparisons can be reproduced and trusted.
The weight-conversion finding also complicates attribution. In a later measurement, GPT-6 Astra found that storing one real weight matrix in a different number format moved it by 0.51% and changed 915,360 of its 1,048,576 values. The source says this was a measurement of conversion, not legal quality or an error rate. It illustrates why a change attributed to a model edit needs to be separated from changes caused by saving the weights.
The published benchmark result remains limited: the edited model answered more of the 10 tasks, but its legal score was still zero. The result does not establish that the edit improved legal performance. The review’s practical contribution was to identify weaknesses in the measurement process and to prompt more qualified reporting.
Products Worth Considering
Sony WH-1000XM6 Wireless Noise-Canceling Headphones, Black
Samsung Galaxy Tab A9+ 11-Inch Android Tablet, 64GB, Graphite
Samsung Galaxy Tab A11+ 11-Inch Tablet, 6GB RAM, 128GB Storage, Expandable, Gray
As an affiliate, we earn on qualifying purchases.
How the Two AI Roles Split
The experiment began on Nov. 4, 2026, with two AI sessions working on the same project. The roles were then separated: Claude Opus 5.5 continued the experiment, while GPT-6 Astra reviewed its implementation. The review was designed to leave the main rig untouched; the Claude session made the fixes.
The underlying experiment used a directional ablation inspired by a MIT CSAIL post on weight-space programming. According to the source, the Claude session identified a “loop direction” in Ornith 1.5 and weakened it in the model weights without training. The article’s initial account described results on a smaller task set; an update dated Nov. 9 says the figures now refer to 10 published Lexassay tasks, with runs available on Lexassay.com.
The source also distinguishes this review from a second model-editing trial. GPT-6 Astra did not produce another edited model for comparison. Its remit was to reproduce faults, measure behavior and review the claims made about the Claude-run experiment.
“Review and validate the other session’s implementation.”
— GPT-6 Astra, in the review report quoted by ThorstenMeyerAI.com
Limits of the Review Evidence
The source says GPT-6 Astra did not rerun every test during its final documentation step, did not act as a fourth grader, and measured weight conversion on a CPU rather than through the production GPU path. Those limits mean the review should not be read as a full independent replication of the experiment or its benchmark results.
It is also unclear from the source how well the 10 published tasks represent broader legal work. The legal score stayed at zero, and no evidence provided here establishes that the edit improved legal reasoning. The review identified implementation faults and corrected claims, but it does not settle the model’s general capabilities.
The source reports that all 12 author tests passed in the second round while four of six reviewer provenance checks failed. It does not provide enough detail in the supplied material to interpret those six checks as a complete audit, or to treat their failure count as a general measure of reliability.
What the Public Runs Can Show
The experiment’s published runs and related materials are available on Lexassay.com, according to the source. Readers can compare the weight-edited model with the edit control and model-card sampling materials, though the article does not say that those materials amount to an independent replication.
The next useful step would be a clearly documented rerun using the corrected code, with the model, sampling settings, task set and weight-conversion path recorded for each run. The source does not announce a date for such a rerun or report further benchmark results. Until more evidence is available, the published task count and the review’s specific findings should be kept separate from claims about broader legal performance.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and benchmark tools used in a Claude Opus 5.5 experiment that edited the weights of Ornith 1.5. It did not run a competing model-editing experiment.
What faults did the review find?
It reported that resumed runs could reuse results from different settings, unknown task IDs could produce empty runs marked successful, a command-line syntax clash broke a real invocation, and weight conversion changed values even at zero edit strength.
Did the edit improve the model’s legal score?
No. The source says the edited model answered all 10 published tasks, compared with 4 before editing, but its legal score remained zero.
Did GPT-6 Astra inspect the hidden benchmark?
The source says the reviewer did not inspect hidden benchmark prompts, topics, answers or reference solutions.
Where can readers find the experiment runs?
ThorstenMeyerAI.com says the published runs and related materials are available on Lexassay.com. The supplied account does not establish that they constitute an independent replication.
Source: ThorstenMeyerAI.com
