Simmerge Learning To Select Merge Operators From Similarity Signals 2026 01 15

SimMerge: Learning to Select Merge Operators from Similarity Signals North Mini Code. Cohere's first model for developers. Learn more

Jan 15, 2026 SimMerge: Learning to Select Merge Operators from Similarity Signals SimMerge is a predictive merge-selection method that uses inexpensive, task-agnostic similarity signals between models to select the best merge operator, subset of models, and merge order, eliminating the expensive merge-and-evaluate loop.

read the paper

Authors

Oliver Bolton, Aakanksha, Arash Ahmadian, Sara Hooker, Marzieh Fadaee, Beyza Ermis

Abstract

Model merging enables multiple large language models (LLMs) to be combined into a single model while preserving performance. This makes it a valuable tool in LLM development, offering a competitive alternative to multi-task training. However, merging can be difficult at scale, as successful merging requires choosing the right merge operator, selecting the right models, and merging them in the right order. This often leads researchers to run expensive merge-and-evaluate searches to select the best merge. In this work, we provide an alternative by introducing SimMerge, a predictive merge-selection method that selects the best merge using inexpensive, task-agnostic similarity signals between models. From a small set of unlabeled probes, we compute functional and structural features and use them to predict the performance of a given 2-way merge. Using these predictions, SimMerge selects the best merge operator, the subset of models to merge, and the merge order, eliminating the expensive merge-and-evaluate loop. We demonstrate that we surpass standard merge-operator performance on 2-way merges of 7B-parameter LLMs, and that SimMerge generalizes to multi-way merges and 111B-parameter LLM merges without retraining. Additionally, we present a bandit variant that supports adding new tasks, models, and operators on the fly. Our results suggest that learning how to merge is a practical route to scalable model composition when checkpoint catalogs are large and evaluation budgets are tight.

Efficiency Generative Models inference Language Models

Related works

Research Tiny Aya: Bridging Scale and Multilingual Depth

Read

Research The Art of Asking: Multilingual Prompt Optimization for Synthetic Data

Read

Research When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs

Read