Scaling Swe Agent Data Collection With Dockerized Environments For Execution

Cerebras Skip to main content

Cerebras Announces First Quarter 2026 Results >>

Nov 24 2025 Scaling SWE Agent Data Collection with Dockerized Environments for Execution Vijay Srinivas

By : Gune S , Sahil Lathiya , APOORV PANDEY , Mritunjai Chandra , Vijay Srinivas , Ganesh Venkatesh Introduction We are focused on building a high-quality platform for agentic flow training, including support for Reinforcement Learning (RL), datasets, and ML recipes. We announced our RL platform a few weeks ago, and this project represents our dataset infrastructure initiative. Why This Matters Challenge : Training effective AI agents for software engineering requires: High Quality Datasets : Diverse Corpus of high quality data which consist of PR title/description, issue title/description, base commit, patches, unit test files and other metadata provided by github Reproducible Environments : Consistent execution environments across different repositories High-Quality Signals : Clear pass/fail signals for learning (FAIL_TO_PASS tests) Scale : Thousands of diverse, real-world software engineering tasks Validation : Verified test outcomes with proper testing

Our Solution : Our Data curation pipeline with the repository sandbox generator provides: Data Curation Pipeline : Built in support for querying repositories to extract their structure, dependencies and various fields required for training SWE agents Automated code execution Infrastructure : Handles complex dependency resolution and environment setup Rich Metadata : Embedded test classifications for training signals Portability : Self-contained Docker images that work anywhere

Impact on Agentic AI Development This infrastructure enables: Supervised Learning : Train models on verified bug fixes with clear before/after states Reinforcement Learning : Provide executable environments with reward signals (test outcomes) Evaluation : Standardized benchmarks for measuring agent performance Research : Reproducible experiments across different approaches

Scalability Vision Our methodology is designed to scale by orders of magnitude : Current Capability : Process 1000+ instances with full automation

We are building the foundation for the largest high-quality dataset for software engineering AI agents. Dataset Collection Methodology To construct a high-quality, diverse corpus of real-world software projects, we sourced repositories from three complementary pipelines: PyPI-linked projects, high-impact GitHub repositories, and multimodal codebases. Each pipeline was designed with strict quality and activity thresholds to ensure that only mature, collaboratively developed projects were included PyPI-Based Repository Instances. We began by identifying repositories associated with PyPI packages, focusing on projects with substantial community engagement. Repositories were required to be predominantly Python (≥60% of their codebase) and show meaningful development activity, with minimum thresholds of 100 pull requests or issues and 100 forks. After applying these filters and a subsequent decontamination pass, this stream contributed approximately 107,000 curated repository instances . GitHub Star-Based Repository Instances. To capture widely adopted and influential projects, we separately collected repositories selected on the basis of GitHub popularity. Only repositories with at least 10,000 stars and clear evidence of active maintenance, at least 100 pull requests or issues and 100 forks were considered. Applying the same ≥60% Python requirement and decontamination produced an additional 50,000 high-quality instances . Multimodal Repository Instances. Finally, to broaden the dataset beyond pure Python projects, we incorporated multimodal repositories that combine Python with other languages or data modalities. Using the same activity criteria (≥100 PRs/issues and ≥100 forks) and a ≥60% Python threshold, we extracted 3,500 instances from a curated set of 700 multimodal repositories .

Together, these pipelines ensure broad coverage across package ecosystems, popular open-source projects, and rich multimodal codebases, resulting in a balanced and rigorously filtered dataset suitable for downstream analysis and model training. Dataset Statistics Below is a quick reference to the key fields captured for each dataset instance. Dataset Comparison & Benchmarks Architecture of Code Execution Environment Code Execution Platform Overivew This code execution platform automates end-to-end differential testing of Python repositories by setting up an isolated workspace, analyzing repository structure and dependencies, generating an optimized dockerfile, and building the environment with automatic LLM-based fixes for build failures. It then performs robust test discovery with iterative auto correction, executes baseline and patched tests in isolated docker runs, and compares results to classify tests as FAIL_TO_PASS or PASS_TO_PASS. A reversal phase rechecks classifications by undoing the main patch, ensuring accuracy. Finally, the system produces comprehensive artifacts which includes logs, docker files, summaries, and detailed JSON results capturing the full build, test, and differential-analysis lifecycle. Interested in the technical details? Reach out to discuss collaboration opportunities in agentic code generation and sandboxed execution for software engineering.

Follow

Get Updates Newsletter Signup

Company About Us Careers Contact Us Investor Relations Website Terms of Use Cookie Policy Other Terms & Policies Service Status Trust Center

News Newsroom In the News Press kit

Insights Customer Spotlight Blog Publications Whitepapers

Performance comparisons are based on third-party benchmarking or internal testing. Observed inference speed improvements versus GPU-based systems may vary depending on workload, configuration, date and models being tested. info@cerebras.ai 1237 E. Arques Ave Sunnyvale, CA 94085

© 2026 Cerebras. All rights reserved.