basetenlabs/axolotl

basetenlabs/axolotl

Description: Go ahead and axolotl questions

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 2

Created: 2024-09-26T23:04:08Z

Pushed: 2024-11-07T15:11:08Z

Default branch: main

Fork: yes

Parent repository: axolotl-ai-cloud/axolotl

Archived: no

README:

Axolotl

!tests !tests-nightly !multigpu-semi-weekly tests

Axolotl is a tool designed to streamline the fine-tuning of various AI models, offering support for multiple configurations and architectures.

Features:

• Train various Huggingface models such as llama, pythia, falcon, mpt
• Supports fullfinetune, lora, qlora, relora, and gptq
• Customize configurations using a simple yaml file or CLI overwrite
• Load different dataset formats, use custom formats, or bring your own tokenized datasets
• Integrated with xformer, flash attention, liger kernel, rope scaling, and multipacking
• Works with single GPU or multiple GPUs via FSDP or Deepspeed
• Easily run with Docker locally or on the cloud
• Log results and optionally checkpoints to wandb or mlflow
• And more!

Table of Contents

• [Axolotl](#axolotl)
• [Table of Contents](#table-of-contents)
• [Axolotl supports](#axolotl-supports)
• [Quickstart ⚡](#quickstart-)
• [Usage](#usage)
• [Advanced Setup](#advanced-setup)
• [Environment](#environment)
• [Docker](#docker)
• [Conda/Pip venv](#condapip-venv)
• [Cloud GPU](#cloud-gpu)
• [Bare Metal Cloud GPU](#bare-metal-cloud-gpu)
• [LambdaLabs](#lambdalabs)
• [GCP](#gcp)
• [Windows](#windows)
• [Mac](#mac)
• [Google Colab](#google-colab)
• [Launching on public clouds via SkyPilot](#launching-on-public-clouds-via-skypilot)
• [Launching on public clouds via dstack](#launching-on-public-clouds-via-dstack)
• [Dataset](#dataset)
• [Config](#config)
• [All Config Options](#all-config-options)
• [Train](#train)
• [Preprocess dataset](#preprocess-dataset)
• [Multi-GPU](#multi-gpu)
• [DeepSpeed](#deepspeed)
• [FSDP](#fsdp)
• [FSDP + QLoRA](#fsdp--qlora)
• [Weights \& Biases Logging](#weights--biases-logging)
• [Special Tokens](#special-tokens)
• [Liger Kernel](#liger-kernel)
• [Inference Playground](#inference-playground)
• [Merge LORA to base](#merge-lora-to-base)
• [Common Errors 🧰](#common-errors-)
• [Tokenization Mismatch b/w Inference \& Training](#tokenization-mismatch-bw-inference--training)
• [Debugging Axolotl](#debugging-axolotl)
• [Need help? 🙋](#need-help-)
• [Badge ❤🏷️](#badge-️)
• [Community Showcase](#community-showcase)
• [Contributing 🤝](#contributing-)
• [Sponsors 🤝❤](#sponsors-)
• [💎 Diamond Sponsors - Contact directly](#-diamond-sponsors---contact-directly)
• [🥇 Gold Sponsors - $5000/mo](#-gold-sponsors---5000mo)
• [🥈 Silver Sponsors - $1000/mo](#-silver-sponsors---1000mo)
• [🥉 Bronze Sponsors - $500/mo](#-bronze-sponsors---500mo)

Axolotl supports

| | fp16/fp32 | lora | qlora | gptq | gptq w/flash attn | flash attn | xformers attn | |-------------|:----------|:-----|-------|------|-------------------|------------|--------------| | llama | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Mistral | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Mixtral-MoE | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | Mixtral8X22 | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | Pythia | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | cerebras | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | btlm | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | mpt | ✅ | ❌ | ❓ | ❌ | ❌ | ❌ | ❓ | | falcon | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❓ | | gpt-j | ✅ | ✅ | ✅ | ❌ | ❌ | ❓ | ❓ | | XGen | ✅ | ❓ | ✅ | ❓ | ❓ | ❓ | ✅ | | phi | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | RWKV | ✅ | ❓ | ❓ | ❓ | ❓ | ❓ | ❓ | | Qwen | ✅ | ✅ | ✅ | ❓ | ❓ | ❓ | ❓ | | Gemma | ✅ | ✅ | ✅ | ❓ | ❓ | ✅ | ❓ | | Jamba | ✅ | ✅ | ✅ | ❓ | ❓ | ✅ | ❓ |

✅: supported ❌: not supported ❓: untested

Quickstart ⚡

Get started with Axolotl in just a few steps! This quickstart guide will walk you through setting up and running a basic fine-tuning task.

Requirements: Python >=3.10 and Pytorch >=2.1.1.

git clone https://github.com/axolotl-ai-cloud/axolotl
cd axolotl

pip3 install packaging ninja
pip3 install -e '.[flash-attn,deepspeed]'

Usage

# preprocess datasets - optional but recommended
CUDA_VISIBLE_DEVICES="" python -m axolotl.cli.preprocess examples/openllama-3b/lora.yml

# finetune lora
accelerate launch -m axolotl.cli.train examples/openllama-3b/lora.yml

# inference
accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \
--lora_model_dir="./outputs/lora-out"

# gradio
accelerate launch -m axolotl.cli.inference examples/openllama-3b/lora.yml \
--lora_model_dir="./outputs/lora-out" --gradio

# remote yaml files - the yaml config can be hosted on a public URL
# Note: the yaml config must directly link to the **raw** yaml
accelerate launch -m axolotl.cli.train https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/examples/openllama-3b/lora.yml

Advanced Setup

Environment

Docker

docker run --gpus '"all"' --rm -it winglian/axolotl:main-latest

Or run on the current files for development:

docker compose up -d

>[!Tip] > If you want to debug axolotl or prefer to use Docker as your development environment, see the [debugging guide's section on Docker](docs/debugging.qmd#debugging-with-docker).

Docker advanced

A more powerful Docker command to run would be this:

docker run --privileged --gpus '"all"' --shm-size 10g --rm -it --name axolotl --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --mount type=bind,src="${PWD}",target=/workspace/axolotl -v ${HOME}/.cache/huggingface:/root/.cache/huggingface winglian/axolotl:main-latest

It additionally:

• Prevents memory issues when running e.g. deepspeed (e.g. you could hit SIGBUS/signal 7 error) through --ipc and --ulimit args.
• Persists the downloaded HF data (models etc.) and your modifications to axolotl code through --mount/-v args.
• The --name argument simply makes it easier to refer to the container in vscode (Dev Containers: Attach to Running Container...) or in your terminal.
• The --privileged flag gives all capabilities to the container.
• The --shm-size 10g argument increases the shared memory size. Use this if you see exitcode: -7 errors using deepspeed.

[More information on nvidia…