arcee-ai/axolotl

arcee-ai/axolotl

Description: Go ahead and axolotl questions

Language: Python

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 1

Created: 2024-07-18T16:29:56Z

Pushed: 2024-07-18T16:32:14Z

Default branch: main

Fork: yes

Parent repository: axolotl-ai-cloud/axolotl

Archived: no

README:

Axolotl

Axolotl is a tool designed to streamline the fine-tuning of various AI models, offering support for multiple configurations and architectures.

Features:

• Train various Huggingface models such as llama, pythia, falcon, mpt
• Supports fullfinetune, lora, qlora, relora, and gptq
• Customize configurations using a simple yaml file or CLI overwrite
• Load different dataset formats, use custom formats, or bring your own tokenized datasets
• Integrated with xformer, flash attention, rope scaling, and multipacking
• Works with single GPU or multiple GPUs via FSDP or Deepspeed
• Easily run with Docker locally or on the cloud
• Log results and optionally checkpoints to wandb or mlflow
• And more!

Table of Contents

• [Introduction](#axolotl)
• [Supported Features](#axolotl-supports)
• [Quickstart](#quickstart-)
• [Environment](#environment)
• [Docker](#docker)
• [Conda/Pip venv](#condapip-venv)
• [Cloud GPU](#cloud-gpu) - Latitude.sh, JarvisLabs, RunPod
• [Bare Metal Cloud GPU](#bare-metal-cloud-gpu)
• [Windows](#windows)
• [Mac](#mac)
• [Google Colab](#google-colab)
• [Launching on public clouds via SkyPilot](#launching-on-public-clouds-via-skypilot)
• [Launching on public clouds via dstack](#launching-on-public-clouds-via-dstack)
• [Dataset](#dataset)
• [Config](#config)
• [Train](#train)
• [Inference](#inference-playground)
• [Merge LORA to Base](#merge-lora-to-base)
• [Special Tokens](#special-tokens)
• [All Config Options](#all-config-options)
• Advanced Topics
• [Multipack](./docs/multipack.qmd)
• [RLHF & DPO](./docs/rlhf.qmd)
• [Dataset Pre-Processing](./docs/dataset_preprocessing.qmd)
• [Common Errors](#common-errors-)
• [Tokenization Mismatch b/w Training & Inference](#tokenization-mismatch-bw-inference--training)
• [Debugging Axolotl](#debugging-axolotl)
• [Need Help?](#need-help-)
• [Badge](#badge-)
• [Community Showcase](#community-showcase)
• [Contributing](#contributing-)
• [Sponsors](#sponsors-)

Axolotl supports

| | fp16/fp32 | lora | qlora | gptq | gptq w/flash attn | flash attn | xformers attn | |-------------|:----------|:-----|-------|------|-------------------|------------|--------------| | llama | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Mistral | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Mixtral-MoE | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | Mixtral8X22 | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | Pythia | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | cerebras | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | btlm | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | mpt | ✅ | ❌ | ❓ | ❌ | ❌ | ❌ | ❓ | | falcon | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | gpt-j | ✅ | ✅ | ✅ | ❌ | ❌ | ❓ | ❓ | | XGen | ✅ | ❓ | ✅ | ❓ | ❓ | ❓ | ✅ | | phi | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | RWKV | ✅ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | | Qwen | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | Gemma | ✅ | ✅ | ✅ | ❓ | ❓ | ✅ | ❓ |

✅: supported ❌: not supported ❓: untested

Quickstart ⚡

Get started with Axolotl in just a few steps! This quickstart guide will walk you through setting up and running a basic fine-tuning task.

Requirements: Python >=3.10 and Pytorch >=2.1.1.

git clone https://github.com/axolotl-ai-cloud/axolotl
cd axolotl

pip3 install packaging ninja
pip3 install -e '.[flash-attn,deepspeed]'

Usage

# preprocess datasets - optional but recommended
CUDA_VISIBLE_DEVICES="" python -m axolotl.cli.preprocess examples/openllama-3b/lora.yml

# finetune lora
accelerate launch -m axolotl.cli.train examples/openllama-3b/lora.yml

# inference
accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \
--lora_model_dir="./outputs/lora-out"

# gradio
accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \
--lora_model_dir="./outputs/lora-out" --gradio

# remote yaml files - the yaml config can be hosted on a public URL
# Note: the yaml config must directly link to the **raw** yaml
accelerate launch -m axolotl.cli.train https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/openllama-3b/lora.yml

Advanced Setup

Environment

Docker

docker run --gpus '"all"' --rm -it winglian/axolotl:main-latest

Or run on the current files for development:

docker compose up -d

>[!Tip] > If you want to debug axolotl or prefer to use Docker as your development environment, see the [debugging guide's section on Docker](docs/debugging.qmd#debugging-with-docker).

Docker advanced

A more powerful Docker command to run would be this:

docker run --privileged --gpus '"all"' --shm-size 10g --rm -it --name axolotl --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --mount type=bind,src="${PWD}",target=/workspace/axolotl -v ${HOME}/.cache/huggingface:/root/.cache/huggingface winglian/axolotl:main-latest

It additionally:

• Prevents memory issues when running e.g. deepspeed (e.g. you could hit SIGBUS/signal 7 error) through --ipc and --ulimit args.
• Persists the downloaded HF data (models etc.) and your modifications to axolotl code through --mount/-v args.
• The --name argument simply makes it easier to refer to the container in vscode (Dev Containers: Attach to Running Container...) or in your terminal.
• The --privileged flag gives all capabilities to the container.
• The --shm-size 10g argument increases the shared memory size. Use this if you see exitcode: -7 errors using deepspeed.

More information on nvidia website

Conda/Pip venv

1. Install python >=3.10

2. Install pytorch stable https://pytorch.org/get-started/locally/

3. Install Axolotl along with python dependencies

pip3 install packaging
pip3 install -e '.[flash-attn,deepspeed]'

4. (Optional) Login to Huggingface to use gated models/datasets.

huggingface-cli login

Get the token at huggingface.co/settings/tokens

Cloud GPU

For cloud GPU providers that support docker images, use `winglian/axolotl-cloud:main-latest`

• on Latitude.sh use this direct link
• on JarvisLabs.ai use this direct link
• on RunPod use this [direct…