🔍 Read the full analysis: FrontierCode: How To Submit An AI Pull Request Maintainers Can Assess on ThorstenMeyerAI.com
Get everyday essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 models on FrontierCode, a benchmark designed to judge whether AI-generated code changes would pass repository-specific checks and maintainer-style review. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; the results show that human review remains necessary, while model costs vary widely.
Artificial Analysis evaluated 23 AI models on FrontierCode, a benchmark that tests whether code changes would meet repository-specific checks and maintainer-style review, rather than merely pass visible tests. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts, leaving substantial room for human review.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, using the benchmark’s 100-task Main set. The leading model was Claude Opus 5.5 (max with fallback), scoring 52.2%. GPT-6.1 Sol (max) placed second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. The scores are not the same as the pass rates: Opus cleared all blocking checks in 57.7% of attempts, while GPT-6.1 Sol did so in 54.7%.
FrontierCode evaluates changes on six dimensions: hidden correctness tests, command checks, tests that must fail against the unmodified code, adaptive tests, limits on files and diff size, and an LLM review against maintainers’ criteria. A failed blocking check gives an attempt a zero, even if other tests pass. Artificial Analysis ran models through the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task; GPT-6 Luna (medium) handled the review step.
The benchmark also makes cost differences visible. Artificial Analysis reported $10.15 per task for the leading Opus configuration and $0.81 for GPT-6.1 Sol. Its identified cost-frontier models included GPT-6 Luna (max), at 41.9% for $0.09 per task, and MiMo-V2.6-Pro, at 45.4% for $0.18. These are reported benchmark costs, not a guarantee of what a particular team would pay in production.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode targets a practical weakness in coding-agent evaluations: a patch can pass tests and still be difficult or unsuitable to merge. By grading scope, test quality, regressions and review criteria alongside correctness, the benchmark attempts to measure more of the work maintainers face when assessing an AI-generated pull request.
The results argue against treating these models as dependable, unattended contributors. Even the top-ranked model cleared the benchmark’s blocking checks in 57.7% of attempts, according to the reported pass-rate metric. That does not mean it would be accepted at that rate in every real repository; the tasks and benchmark rules are a specific evaluation. But it does show why teams should retain review, testing and repository-specific safeguards rather than relying on a leaderboard score alone.
Cost also affects the choice. The gap between GPT-6.1 Sol and Opus is 3.1 score points at the reported settings, alongside a large difference in cost per task. For engineering teams, the relevant decision is not simply which model ranks highest, but whether its additional performance justifies its price for the work being delegated.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was built by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July with rules against searching online for solutions. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source report says maintainers spent more than 40 hours on each task. The full benchmark has 150 tasks, while Artificial Analysis used the 100-task Main set.
Tasks are private, according to the source report, to reduce the chance that models encountered their solutions during training. The benchmark also penalizes attempts flagged by Cognition’s trajectory scanner for looking up solutions. Oversized diffs, edits outside relevant files and unhelpful tests can reduce scores. Passing all blocking checks produces a score between 70% and 100% under the benchmark’s scoring rules, with quality checks affecting the final result.
The leaderboard’s top nine models scored between 43.2% and 52.2%. MiMo-V2.6-Pro led the listed open-weight models at 45.4%, followed by DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. Results also differed from a general intelligence ranking cited by the source: DeepSeek V4.1 Flash ranked seventh on FrontierCode, while Grok 4.7 scored 33.8% and ranked 18th in the comparison. This illustrates that general model rankings do not necessarily predict performance on repository-level coding tasks.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The published figures cover one benchmark setup: Artificial Analysis’s 100-task Main set, its chosen agent harness and model configurations, and a particular review model. The available source material does not establish how well those scores predict acceptance rates across different teams, repositories or production workflows. Nor does a benchmark pass rate say whether a change is safe to merge without a human checking its context.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the current table cannot compare their performance on this benchmark. The source report also does not provide results for every model-and-cost configuration or explain how all task-level outcomes vary, so the headline averages should not be read as a complete account of reliability.
machine learning code testing tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Watch for More Model Results
The next useful development will be additional FrontierCode evaluations, particularly for models that were missing from the October 13 results. Readers can also compare future results with the current scores and cost figures, while checking whether the benchmark tasks, scoring rules or evaluation setup have changed. Any such changes would affect how directly new results can be compared with this run.
For teams using coding agents now, the reported results support a measured approach: treat generated changes as proposals for review, run repository-specific checks, and examine scope and tests before merging. FrontierCode offers one structured way to compare models, but the source material does not report that any model can replace maintainers’ judgment in a live project.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated code changes against six types of checks, including hidden tests, regression and command checks, test quality, change scope, and an LLM review based on maintainers’ criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the reported results with a 52.2% FrontierCode score. Its pass rate for all blocking checks was 57.7%.
Does the top score mean an AI pull request is ready to merge?
No. The benchmark is an evaluation, not a guarantee of acceptance in a real repository. Its results leave room for failed checks, and teams still need to apply their own tests and review.
Which model offered the lowest reported task cost?
Among the cost-frontier models listed in the source report, GPT-6 Luna (max) scored 41.9% at a reported $0.09 per task. The figure reflects the benchmark setup and may not match costs in other workflows.
Which models were missing from the results?
As of October 13, 2026, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
