Which AI Configuration Should Run A News Pipeline? I Tested 71

🔍 Read the full analysis: Which AI Configuration Should Run A News Pipeline? I Tested 71 on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get everyday essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer tested 71 AI configurations on news articles for his sites and reports that GPT-6 Astra scored highest, but GPT-6 Luna was chosen for production because it delivered consistent results at a far lower reported cost. The test, published by ThorstenMeyerAI.com on October 13, also found two backup models produced fabricated quotes or output the pipeline could not use.

Meyer says the evaluation ran from September 8 to October 13, 2026, using his production prompt and real news sources. The configurations included API and subscription models, free-tier services and open-weight models run on his hardware. Each source was turned into an English and a German article, then graded by two AI judges against the source material.

In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages, with a source-fidelity score of 5.00. It averaged 3.93 across 12 articles in a further robustness run, according to Meyer. He reports Astra cost about $0.12 per article and took about 50 seconds. GPT-6 Luna scored 3.67 in the robustness run, with source fidelity of 4.71, and every article it generated was valid under the pipeline’s repair rules. Its reported cost was about $0.0012 per article and generation time about 17 seconds.

Meyer says he kept Luna as the production model because of that cost and performance balance. He also tested Luna at six reasoning-effort settings. The reported scores varied, but he judged the differences to be within the test’s noise; the maximum setting cost about 12 times as much and took about 12 times as long as the no-effort setting, without a reliable quality gain.

Cost and Reliability Set the Choice

The results address a practical decision for publishers using automated writing: the highest-scoring model may not be the one suited to routine production. In Meyer’s test, Astra led on writing quality, but its reported per-article cost was roughly 100 times Luna’s. For high-volume publishing, that difference can shape whether a small improvement in scores is worth the expense and additional time.

The reported findings also point to a separate operational risk: fallback models can affect accuracy when the primary service fails. Meyer says GPT-4.1 nano received a source-fidelity score of 1.00 and generated unsupported quotations, while a local gpt-oss-120b model produced no valid articles in four attempts. Those results led him to remove both from the route. They are findings from this particular test, not evidence that the models will behave identically across other prompts or systems.

How the News-Writing Test Worked

Meyer says the test used two sources: a detailed product filing of 6,340 characters and a short political news item of 619 characters. He described the shorter source as more representative of much of his production work. The models had to produce articles in English and German, with the judges scoring source fidelity, fabrication, clarity about what was known, news value, structure and polish; German language quality was also assessed.

The published method included a robustness run of 10 additional articles for stronger configurations, mostly based on short sources. Meyer reports 95% bootstrap intervals for results and says that, with only two articles in the initial test, differences below about 0.3 points should be treated as noise. He used whether an article remained valid after the same repair step as production as a pass-or-fail usability measure.

The test also compared models beyond the two leading GPT configurations. Meyer reports that Claude Opus 5.5 scored 3.46 in English and 3.39 in German in the standard test, while Claude Sonnet 5.5 scored 3.67 and 3.61. He cautions through the comparison that performance on coding benchmarks does not establish performance on news writing; each task requires its own evaluation.

“The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, ThorstenMeyerAI.com

Limits of the Published Results

The results come from one publisher’s prompt, pipeline and source selection, and the supplied account does not provide the full article set or independently verified score sheets. The initial comparison used two sources, while the larger robustness run covered 10 additional articles for stronger configurations. The reported rankings may not hold for different beats, languages, editorial standards or prompts.

Meyer says two AI judges assessed the work and that their intervals help show uncertainty, but the article does not establish how closely those scores match human editors’ judgments. It is also unclear from the supplied material how the models would perform across a longer production period, how often the primary model would fail, or whether the removed fallbacks have since been replaced. The reported costs and timings are tied to Meyer’s setup and should not be treated as universal prices or service guarantees.

Luna Remains in Production

Meyer says GPT-6 Luna remains the production choice, with the no-reasoning-effort setting retained. He reported that the two weak backup models were removed on October 7. The supplied account does not name a replacement fallback or describe the current failover arrangement, so how the pipeline will handle a Luna outage remains unclear.

The next useful evidence would be performance over a larger, continuing sample of real articles, including failures and human editorial reviews. Meyer’s published results establish his choice for this pipeline at the time of publication, but they do not settle which configuration will work best for other publishers or future model versions.

Key Questions

Which configuration scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in Meyer’s standard test. It averaged 3.93 across 12 articles in the reported robustness run.

Why did Meyer choose GPT-6 Luna instead?

Meyer reports that Luna scored 3.67 in the robustness run and produced valid articles throughout the test. He says it cost about $0.0012 per article, compared with about $0.12 for Astra, making it his production choice.

Did higher reasoning effort improve Luna’s articles?

Not reliably in Meyer’s results. Scores at the six effort settings were close enough that he treated the differences as noise; the maximum setting was reported to cost and take about 12 times as much as the no-effort setting.

What problems did the fallback models produce?

Meyer says GPT-4.1 nano generated unsupported quotations and received a source-fidelity score of 1.00. He reports that gpt-oss-120b produced no valid articles in four attempts and also generated invented quotations. Both were removed from the route on October 7.

Can these results determine the best model for every newsroom?

No. The test used Meyer’s own production prompt, two news sources, two languages and his pipeline’s validity rules. The reported results are specific to that setup and do not establish how the configurations will perform for other publishers.

Source: ThorstenMeyerAI.com