mistralai/FastChat-release

mistralai/FastChat-release

Description: An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

License: Apache-2.0

Stars: 50

Forks: 9

Open issues: 0

Created: 2023-09-27T15:01:17Z

Pushed: 2023-10-02T21:11:01Z

Default branch: main

Fork: yes

Parent repository: lm-sys/FastChat

Archived: yes

README:

FastChat

| **Demo** | **Discord** | **X** |

FastChat is an open platform for training, serving, and evaluating large language model based chatbots. The core features include:

• The training and evaluation code for state-of-the-art models (e.g., Vicuna).
• A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs.

News

• [2023/08] 🔥 We released Vicuna v1.5 based on Llama 2 with 4K and 16K context lengths. Download [weights](#vicuna-weights).
• [2023/08] 🔥 We released LongChat v1.5 based on Llama 2 with 32K context lengths. Download [weights](#longchat).
• [2023/07] We released Chatbot Arena Conversations, a dataset containing 33k conversations with human preferences. Download it here.

More

• [2023/06] We introduced MT-bench, a challenging multi-turn question set for evaluating chatbots. Check out the blog post.
• [2023/06] We introduced LongChat, our long-context chatbots and evaluation tools. Check out the blog post.
• [2023/05] We introduced Chatbot Arena for battles among LLMs. Check out the blog post.
• [2023/03] We released Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. Check out the blog post.

Contents

• [Install](#install)
• [Model Weights](#model-weights)
• [Inference with Command Line Interface](#inference-with-command-line-interface)
• [Serving with Web GUI](#serving-with-web-gui)
• [API](#api)
• [Evaluation](#evaluation)
• [Fine-tuning](#fine-tuning)
• [Citation](#citation)

Install

Method 1: With pip

pip3 install "fschat[model_worker,webui]"

Method 2: From source

1. Clone this repository and navigate to the FastChat folder

git clone https://github.com/lm-sys/FastChat.git
cd FastChat

If you are running on Mac:

brew install rust cmake

2. Install Package

pip3 install --upgrade pip # enable PEP 660 support
pip3 install -e ".[model_worker,webui]"

Model Weights

Vicuna Weights

Vicuna is based on LLaMA and should be used under LLaMA's model license.

You can use the commands below to start chatting. It will automatically download the weights from Hugging Face repos. See more command options and how to handle out-of-memory in the "Inference with Command Line Interface" section below.

NOTE: `transformers>=4.31` is required for 16K versions.

| Size | Chat Command | Hugging Face Repo | | --- | --- | --- | | 7B | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 | lmsys/vicuna-7b-v1.5 | | 7B-16k | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5-16k | lmsys/vicuna-7b-v1.5-16k | | 13B | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5 | lmsys/vicuna-13b-v1.5 | | 13B-16k | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5-16k | lmsys/vicuna-13b-v1.5-16k | | 33B | python3 -m fastchat.serve.cli --model-path lmsys/vicuna-33b-v1.3 | lmsys/vicuna-33b-v1.3 |

Old weights: see [docs/vicuna_weights_version.md](docs/vicuna_weights_version.md) for all versions of weights and their differences.

LongChat

We release LongChat models under LLaMA's model license.

| Size | Chat Command | Hugging Face Repo | | --- | --- | --- | | 7B | python3 -m fastchat.serve.cli --model-path lmsys/longchat-7b-32k-v1.5 | lmsys/longchat-7b-32k |

FastChat-T5

You can use the commands below to chat with FastChat-T5. It will automatically download the weights from Hugging Face repos.

| Size | Chat Command | Hugging Face Repo | | --- | --- | --- | | 3B | python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0 | lmsys/fastchat-t5-3b-v1.0 |

Inference with Command Line Interface

(Experimental Feature: You can specify --style rich to enable rich text output and better text streaming quality for some non-ASCII content. This may not work properly on certain terminals.)

Supported Models

FastChat supports a wide range of models, including LLama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MTP, OpenAssistant, RedPajama, StableLM, WizardLM, and more.

See a complete list of supported models and instructions to add a new model [here](docs/model_support.md).

Single GPU

The command below requires around 14GB of GPU memory for Vicuna-7B and 28GB of GPU memory for Vicuna-13B. See the ["Not Enough Memory" section](#not-enough-memory) below if you do not have enough memory. --model-path can be a local folder or a Hugging Face repo name.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3

Multiple GPUs

You can use model parallelism to aggregate GPU memory from multiple GPUs on the same machine.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3 --num-gpus 2

Tips: Sometimes the "auto" device mapping strategy in huggingface/transformers does not perfectly balance the memory allocation across multiple GPUs. You can use --max-gpu-memory to specify the maximum memory per GPU for storing model weights. This allows it to allocate more memory for activations, so you can use longer context lengths or larger batch sizes. For example,

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3 --num-gpus 2…