📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite its impressive technical architecture. This performance highlights a significant challenge in the European sovereign-LLM strategy—scale alone may not produce country-knowledge depth.

The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research infrastructure, built a 7-billion-parameter model from scratch, trained on a dataset of 2.5 trillion tokens, roughly half of which were Italian. The project aimed to demonstrate that a domestic, open-weights LLM could outperform multilingual models on Italian benchmarks and serve as a model for European AI sovereignty.

However, despite its technical achievements, Minerva-3B scored just 4.9% on the INVALSI Italian academic-content tests, a near-chance result that indicates a disconnect between training data scale and actual language understanding in complex tasks. Researchers noted that while dataset composition matters, overall size and parameters are more critical for handling complex language tasks.

This empirical finding suggests that the substantial investment in native-language data and infrastructure may still be insufficient at the current parameter scales, raising questions about the optimal investment levels needed for meaningful country-specific AI capabilities. The results challenge the assumption that larger, native-language models automatically translate into deeper language and knowledge understanding.

Implications for European Sovereign-LLM Strategies

The performance gap revealed by Minerva-3B indicates that European efforts to develop domestic LLMs cannot rely solely on scale and native-language data. It underscores the need for a reevaluation of investment levels and architectural strategies to achieve genuine country-specific knowledge and capabilities. This finding has broad implications for policymakers and AI developers across Europe, emphasizing that scale alone may not suffice for complex language understanding and academic competence.

European Sovereign-LLM Development Approaches and Challenges

Italy’s Minerva project represents a significant investment in building a large-scale, open-weights LLM from scratch, contrasting with approaches like Portugal’s AMÁLIA, which layered specialization onto multilingual foundations. Minerva trained on a dataset of 2.5 trillion tokens, with approximately 50% Italian content, and was supported by Italy’s national research infrastructure and funding programs.

Despite these efforts, Minerva’s performance on academic benchmarks has been disappointing, prompting a broader discussion on the effectiveness of scaling and data composition strategies in European sovereign AI projects. The project’s empirical results suggest that current scaling may not be sufficient to produce truly country-specific language understanding, challenging assumptions underlying many national AI initiatives.

This situation underscores a broader debate within the European AI community about the optimal balance between data, model size, and architectural design to meet national and linguistic needs effectively.

“The results suggest that dataset composition and size are critical, but current parameter scales still fall short for complex language tasks in specific languages.”

— Research team, Minerva project

Unresolved Questions About Scale and Effectiveness

It remains unclear what the optimal scale and investment thresholds are for achieving meaningful country-specific language understanding. The performance of Minerva-3B raises questions about whether larger models or different training strategies could overcome current limitations, but definitive answers are still forthcoming.

Next Steps for European Sovereign AI Development

The Minerva team is continuing to iterate on their methodology, including ongoing experiments in continual training and larger models. Policymakers and researchers are expected to reassess investment strategies and architectural approaches based on these empirical insights, aiming to bridge the gap between technical performance and real-world language understanding.

Key Questions

Why did Minerva-3B perform poorly on Italian academic tests?

Despite extensive training data and a large model size, the empirical results suggest that scale alone is insufficient for complex language understanding. The model may lack the nuanced knowledge and reasoning capabilities needed for academic content, indicating a need for different training approaches or larger models.

Does this mean European sovereign models are ineffective?

Not necessarily. The results highlight the challenges of scale and data composition, but ongoing research aims to identify the most effective strategies. It underscores that achieving deep language and domain knowledge may require more than just larger datasets and models.

What are the implications for future AI investments in Europe?

The findings suggest that European AI initiatives should consider increasing investment in model scale, data quality, and architectural innovation. Simply scaling up current models may not suffice to meet national language and knowledge needs.

Source: ThorstenMeyerAI.com