microsoft/AutoAdapt

microsoft/AutoAdapt

Description: AutoAdapt: An Automated Domain Adaptation Framework for Large Language Models

Language: Python

License: MIT

Stars: 28

Forks: 2

Open issues: 5

Created: 2026-02-16T09:16:18Z

Pushed: 2026-06-13T20:38:07Z

Default branch: main

Fork: no

Archived: no

README:

⚡ AutoAdapt

This is the official implementation of AutoAdapt: An Automated Domain Adaptation Framework for Large Language Models

Large language models (LLMs) excel in open domains but struggle in specialized settings with limited data and evolving knowledge. Existing domain adaptation practices rely heavily on manual trial-and-error processes, incur significant hyperparameter complexity, and are highly sensitive to data and user preferences, all under the high cost of LLM training. Moreover, the interactions and transferability of hyperparameter choices across domains remain poorly understood, making adaptation gains uncertain even with substantial effort. To solve these challenges, we present AutoAdapt, a novel end-to-end automated framework for efficient and reliable LLM domain adaptation. AutoAdapt leverages curated knowledge bases from literature and open-source resources to reduce expert intervention. To narrow the search space, we design a novel multi-agent debating system in which proposal and critic agents iteratively interact to align user intent and incorporate data signals and best practices into the planning process. To optimize hyperparameters under tight budgets, we propose AutoRefine, a novel LLM-based surrogate that replaces costly black-box search. Across 10 tasks, AutoAdapt achieves a 25% average accuracy improvement over state-of-the-art Automated Machine Learning baselines with minimal optimization overhead.

Installation and Quick Start

We recommend using a conda environment:

conda create --name autoadapt python=3.12.3

To install AutoAdapt, execute the following command, after cloning the repository:

pip install .

Usage

AutoAdapt supports two ways of usage: 1. A Command Line Interface (CLI) 2. A React-based UI that runs locally in your browser

Choose the option that best fits your needs.

Option 1: Use AutoAdapt via CLI

Follow the steps below to configure and run AutoAdapt:

1. Navigate to the app/sample directory and create a data subdirectory. Place your training dataset, test dataset, and RAG context files (if needed) within this folder.

2. Update the configuration file located at app/sample/conf/config.yaml with the following:

• Path to the project directory (general.project_path)
• Paths to the respective datasets and context documents (data.dataset_name_or_path, data.test_dataset_name_or_path)
• Column nomenclature specifications (data.input_column, data.response_column)
• Other optional configurations

3. Set the Hugging Face access token, OpenAI API Key and Serp API Key:

export HF_ACCESS_TOKEN="your_hf_token"
export API_KEY="your_openai_key"
export SERPAPI_KEY="your_serpapi_key"

3. Customize the configuration file according to your requirements.

4. (Optional) Update the app/sample/conf/user_preferences.md file to specify any user preferences.

5. (Optional) Update/Add an evaluation function specific to your task in the src/autoadapt/utils/template.py file and specify the function name using the evaluation.eval_func config.

6. Navigate to the application directory (app/sample) and execute the following command to start AutoAdapt:

python3 run.py

Option 2: Use AutoAdapt via the UI

Follow the steps below to configure and run AutoAdapt UI:

1. Ensure that Node.js and npm is installed.

2. Set the Hugging Face access token, OpenAI API Key and Serp API Key:

export HF_ACCESS_TOKEN="your_hf_token"
export API_KEY="your_openai_key"
export SERPAPI_KEY="your_serpapi_key"

3. Navigate to app/react and run the setup.sh file:

cd app/react
bash setup.sh

4. Once the servers have started, run the following command:

bash start.sh

5. Open browser and navigate to http://localhost:8080.

Note: The templates provided in src/autoadapt/template can also be directly used by navigating to the required directory (e.g. src/autoadapt/template/sft) and changing the config file (e.g. src/autoadapt/template/sft/conf/config.yaml) within the directory, for domain-adaptation of LLM on hyperparameters chosen by the user.

Folder Structure

AutoAdapt/
├── app/ # Application entry points and data
│ ├── data/
│ │ └── models_data_database/ # Model and dataset metadata (JSON)
│ ├── react/ # React-based UI
│ │ ├── backend/ # FastAPI backend server
│ │ │ └── conf/ # Backend configuration
│ │ ├── frontend/ # React + Vite frontend
│ │ │ ├── public/
│ │ │ └── src/
│ │ ├── setup.sh # UI setup script
│ │ └── start.sh # UI start script
│ └── sample/ # CLI sample application
│ ├── conf/ # Sample config (config.yaml, user_preference.md)
│ ├── run.py # Main CLI entry point
│ ├── run_plan.py # Plan-only execution
│ └── run_dashboard.sh # Dashboard launcher
│
├── src/
│ ├── autoadapt/ # Core AutoAdapt package
│ │ ├── cli.py # CLI interface
│ │ ├── adapt/ # Adaptation pipeline orchestration
│ │ │ ├── adapt_pipeline.py # End-to-end adaptation pipeline
│ │ │ ├── adapt_step.py # Individual adaptation step
│ │ │ ├── autoadapt_executor.py # AutoAdapt executor
│ │ │ ├── base_executor.py # Base executor class
│ │ │ ├── extract_dataset_stats.py# Dataset statistics extraction
│ │ │ ├── pipeline.py # Pipeline definition
│ │ │ └── pipeline_executor.py # Pipeline execution engine
│ │ ├── agents/ # Multi-agent system
│ │ │ ├── aggregator_agent.py # Aggregates agent outputs
│ │ │ ├── base_agent.py # Base agent class
│ │ │ ├── base_worker.py # Base worker class
│ │ │ ├── best_practice_agent.py # Best practices retrieval agent
│ │ │ ├── data_agent.py # Data analysis agent
│ │ │ ├── knowledge_retriever_agent.py # Knowledge retrieval agent
│ │ │ ├── messages.py # Agent message definitions
│ │ │ ├── research_agent.py # Research/web search agent
│ │ │ ├── user_preference_agent.py# User preference parsing agent
│ │ │ ├── best_practices/ # Best practice retrieval utilities
│ │ │ └── knowledge_retriever/ # Knowledge retrieval utilities
│ │ ├── config/ # Configuration and knowledge bases
│ │ │ ├── knowledge.md # Domain adaptation knowledge base
│ │ │ ├──...