📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Recent AI progress indicates engineering tasks in AI development are now largely automated, while research remains less automatable. This could accelerate AI innovation and alter research workflows.
Recent evidence suggests that AI systems have automated the majority of engineering tasks involved in AI development, with research remaining the primary residual challenge, marking a significant shift in AI capabilities.
According to Thorsten Meyer’s analysis of Jack Clark’s recent essay, six key benchmarks measuring AI’s ability in core AI research skills are approaching saturation, indicating that AI can now automate substantial portions of engineering work. For example, the CORE-Bench, which tests research reproduction, has improved from 21.5% to 95.5% over fifteen months, with some authors declaring it ‘solved.’ Similarly, the MLE-Bench, assessing performance on Kaggle competitions, has risen from 16.9% to 64.4% within sixteen months, reaching a level comparable to mid-tier human practitioners. These advancements suggest that the bottleneck in AI research is shifting from engineering to the research process itself, which remains less automatable.
Clark’s analysis emphasizes that while engineering tasks such as reproducing experiments and optimizing kernels are now highly automatable, the more creative and conceptual aspects of research—such as hypothesis generation and theoretical innovation—are less amenable to automation. The progression across multiple independent benchmarks indicates a rapid approaching of saturation points, implying that much of the engineering work in AI development may soon be handled by AI systems, potentially reducing research timelines and costs.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
AI Workflow Automation for Bloggers: Build a Simple Content System to Research, Write, Optimize, and Repurpose Posts Faster with AI and No-Code Tools (AI Toolkit for Bloggers 2026 Book 8)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
AI Workflow Tools for Researchers & Analysts: Automating Literature Reviews, Summaries, and Hypothesis Generation with ChatGPT, Claude, and Perplexity
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
Applying AI in Learning and Development: From Platforms to Performance
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications for AI Development and Research Pace
This shift means AI development could accelerate significantly, as engineering bottlenecks diminish. With engineering automation nearing completion, research—traditionally the slower, more uncertain phase—may become the new frontier. This could lead to faster innovation cycles, reduced costs, and a potential redefinition of how AI research is conducted, with AI systems possibly taking on roles in hypothesis formulation and experimental design. However, it also raises questions about the future of AI research labor and the nature of scientific discovery.
Progress of AI in Core Research Skills Over Time
Prior to 2024, AI capabilities in research-related tasks were progressing steadily but slowly. The recent surge, highlighted by the rapid improvements in benchmarks like CORE-Bench and MLE-Bench, reflects a structural shift where AI systems are now approaching or surpassing human-level performance in specific engineering tasks. This trend aligns with broader developments in AI, such as the increasing sophistication of models in code generation, kernel design, and experiment reproduction, driven by continuous improvements in model architectures and training data.
Clark’s essay and Meyer’s analysis suggest that these capabilities are approaching saturation, with some benchmarks already considered ‘solved.’ The pattern indicates that the remaining challenge is less about technical capability and more about the nature of research itself—particularly the creative and conceptual aspects that are harder to automate.
“Clark’s conclusion is correct and possibly understated for engineering. The residual research question is real but may be less binding than the framing suggests.”
— Thorsten Meyer
Uncertainties About Research Automation Limits
It remains unclear how much of the research process—such as hypothesis generation, theoretical innovation, and conceptual breakthroughs—can be automated. While engineering tasks are nearing full automation, the creative and abstract aspects of research are less certain to follow suit in the near term. The exact timeline and extent of automation in these areas are still under investigation and debate among experts.
Next Steps in AI Automation and Research Development
Researchers and institutions will likely focus on developing AI tools that assist or automate more complex research activities, including hypothesis formulation and experimental design. Monitoring the progress of benchmarks and real-world applications will be crucial to understanding how quickly research itself becomes more automatable. Additionally, ethical and policy considerations around AI-driven research will gain importance as capabilities expand.
Key Questions
What are the main engineering tasks now automated by AI?
Reproducing research experiments, optimizing computational kernels, and generating production-ready code are among the tasks now largely handled by AI systems.
Does automation mean AI will replace human researchers?
While AI automates many engineering tasks, the creative, conceptual, and theoretical aspects of research remain less automatable, so human researchers will continue to play a vital role.
How soon might research itself become fully automatable?
Current evidence suggests that engineering automation is nearing completion, but full automation of research, especially its creative components, may still be years away and depends on future breakthroughs.
What are the risks of highly automated AI research?
Potential risks include reduced diversity of scientific ideas, over-reliance on AI-generated hypotheses, and ethical concerns about autonomous research processes. These require careful oversight and regulation.
Source: ThorstenMeyerAI.com
