Verification Limits Code Llm Training 2025 09 26

Verification Limits Code LLM Training North Mini Code. Cohere's first model for developers. Learn more

Sep 26, 2025 Verification Limits Code LLM Training Synthetic data generation for code models faces a "verification ceiling" due to verifier limitations. Richer test suites, relaxed pass thresholds, and diverse solutions improve performance. Calibrated verification with challenging problem-solution pairs can overcome this ceiling.

Read the paper

Authors

Srishti Gureja, Elena Tommasone, Jingyi He, Sara Hooker, Matthias Gallé, Marzieh Fadaee

Abstract

Large language models for code generation increasingly rely on synthetic data, where both problem solutions and verification tests are generated by models. While this enables scalable data creation, it introduces a previously unexplored bottleneck: the verification ceiling, in which the quality and diversity of training data are fundamentally constrained by the capabilities of synthetic verifiers. In this work, we systematically study how verification design and strategies influence model performance. We investigate (i) what we verify by analyzing the impact of test complexity and quantity: richer test suites improve code generation capabilities (on average +3 pass@1), while quantity alone yields diminishing returns, (ii) how we verify by exploring relaxed pass thresholds: rigid 100% pass criteria can be overly restrictive. By allowing for relaxed thresholds or incorporating LLM-based soft verification, we can recover valuable training data, leading to a 2-4 point improvement in pass@1 performance. However, this benefit is contingent upon the strength and diversity of the test cases used, and (iii) why verification remains necessary through controlled comparisons of formally correct versus incorrect solutions and human evaluation: retaining diverse correct solutions per problem yields consistent generalization gains. Our results show that Verification as currently practiced is too rigid, filtering out valuable diversity. But it cannot be discarded, only recalibrated. By combining calibrated verification with diverse, challenging problem-solution pairs, we outline a path to break the verification ceiling and unlock stronger code generation models.

Code Data Efficiency Verification

Related works

Research To Code, or Not To Code? Exploring Impact of Code in Pre-training

Read

Research Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement

Read

Research Policy Primer - Efficient AI

Read