openbmb/UltraX-0.6B-Preview

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

📜 Paper | 💻 Code | 🤗 Datasets | 📦 UltraData Collection

English | 中文

📚 Introduction

UltraX is a function-calling refinement framework for large-scale pre-training data. Instead of end-to-end text rewriting, UltraX trains a lightweight refinement model to predict structured editing operations — including insertion, deletion, and modification — which are then deterministically executed on the original text.

This repository provides the UltraX refinement models, which are trained via full-parameter SFT to predict cleaning function calls for web-crawled text:

| Model | Description | |-------|-------------| | UltraX | Default model: instruction-guided with edit-weighted sampling (main experiment) | | UltraX-No-Instruction | Ablation variant: trained without system instruction | | UltraX-Preservation-Weighted | Ablation variant: trained with preservation-weighted sampling (60% keep_all) |

📢 News

• [2026.07.13] UltraX codebase, refinement model, and refined datasets are now available on GitHub and Hugging Face. 🚀🚀🚀
• [2025.07.10] UltraX technical report is available on arXiv. 🔥🔥🔥

💡 Highlights

• Function-Calling Refinement: Instead of end-to-end text rewriting, UltraX predicts structured editing operations (keep_all, remove_all, remove_lines, replace_str, add_line), enabling fine-grained instance-level editing with deterministic execution.
• LAM + DCR Pipeline: Line Alignment and Mapping (LAM) aligns original and refined text at line level, while Dynamic Context Replacement (DCR) converts character-level edits into reliable replace_str operations with unique context anchoring.
• Lightweight & Efficient: The refinement model is a standard SFT model that predicts structured function calls, significantly cheaper than end-to-end LLM rewriting while achieving superior performance.

🔬 Pipeline Overview

🔍 Model Details

All models are trained with the following configuration:

| Setting | Value | |---------|-------| | Training Framework | ms-swift with DeepSpeed ZeRO3 | | Training Type | Full-parameter SFT | | Max Sequence Length | 20,480 tokens | | Learning Rate | 3e-5 (cosine decay, min lr 3e-6) | | Warmup Ratio | 0.03 | | Precision | bfloat16 | | Hardware | 8x GPU |

Model Variants

• UltraX (Instruction-Guided, Edit-Weighted): The default configuration. The model is trained with a system instruction that defines the editing operations and cleaning rules, using edit-weighted sampling.

• UltraX-No-Instruction: Trained without system instruction. Demonstrates that programmatic refinement brings clear gains even without explicit task protocol (Avg: 45.73 vs Raw 45.08), though instruction guidance further improves performance (Avg: 46.14).

• UltraX-Preservation-Weighted: Trained with preservation-weighted sampling where keep_all accounts for 60% of training data. Shows stronger performance on certain tasks (ARC-C, ARC-E, OBQA, SciQ), demonstrating the trade-off between editing aggressiveness and content preservation.

📈 Evaluation Results

Performance comparison on FineWeb (20B tokens, 1B MiniCPM, 10 benchmarks, zero-shot):

🔧 Function Space

The model predicts operations from the following function space:

| Function | Description | |----------|-------------| | keep_all() | Document needs no modification | | remove_all() | Entire document is valueless (e.g., error pages, login walls) | | remove_lines(start, end) | Remove consecutive lines from start to end (inclusive) | | replace_str(line, old, new) | Replace a substring within a specific line | | add_line(base, sub_idx, content) | Insert a new line near the base position |

Input/Output Format

Input: Text with line number markers (`` prefix per line), preceded by a system instruction.

Output: A sequence of function calls, one per line. Example:

remove_lines(1, 3)
replace_str(5, 'Click here to subscribe', '')
remove_lines(12, 15)

🚀 Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("openbmb/UltraX-0.6B-Preview")
tokenizer = AutoTokenizer.from_pretrained("openbmb/UltraX-0.6B-Preview")

❤️ Acknowledgements

• The refinement model is trained using ms-swift.
• UltraX builds upon insights from ProX and RefineX.

Thanks for their awesome work! Open-source contributions make UltraX possible! 🙌

📖 Citation

If you find our work useful, please consider citing:

@misc{ultrax2026,
title={UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing},
author={Xinlong Zhao and Dongsheng Liu and Hengyu Zhao and Zixuan Fu and Zheng Wang and Jie Cai and Jie Zhou and Qiang Ma and Xuanhe Zhou and Xu Han and Yudong Wang and Zhiyuan Liu},
year={2026},
eprint={2607.08646},
archivePrefix={arXiv},
primaryClass={cs.CL},
}

📜 License

This project is licensed under the [Apache 2.0](./LICENSE) license.