mistralai/mistral-inference

mistralai/mistral-inference

Description: Official inference library for Mistral models

Language: Jupyter Notebook

License: Apache-2.0

Stars: 10813

Forks: 1052

Open issues: 175

Created: 2023-09-27T13:05:24Z

Pushed: 2026-04-20T10:10:46Z

Default branch: main

Fork: no

Archived: yes

README:

Mistral Inference

This repository contains minimal code to run Mistral models.

Blog 7B: https://mistral.ai/news/announcing-mistral-7b/\ Blog 8x7B: https://mistral.ai/news/mixtral-of-experts/\ Blog 8x22B: https://mistral.ai/news/mixtral-8x22b/\ Blog Codestral 22B: https://mistral.ai/news/codestral \ Blog Codestral Mamba 7B: https://mistral.ai/news/codestral-mamba/ \ Blog Mathstral 7B: https://mistral.ai/news/mathstral/ \ Blog Nemo: https://mistral.ai/news/mistral-nemo/ \ Blog Mistral Large 2: https://mistral.ai/news/mistral-large-2407/ \ Blog Pixtral 12B: https://mistral.ai/news/pixtral-12b/ Blog Mistral Small 3.1: https://mistral.ai/news/mistral-small-3-1/

Discord: https://discord.com/invite/mistralai\ Documentation: https://docs.mistral.ai/\ Guardrailing: https://docs.mistral.ai/usage/guardrailing

Installation

Note: You will use a GPU to install mistral-inference, as it currently requires xformers to be installed and xformers itself needs a GPU for installation.

PyPI

pip install mistral-inference

Local

cd $HOME && git clone https://github.com/mistralai/mistral-inference
cd $HOME/mistral-inference && poetry install .

Model download

Direct links

| Name | Download | md5sum | |-------------|-------|-------| | 7B Instruct | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar | 80b71fcb6416085bcb4efad86dfb4d52 | | 8x7B Instruct | https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar (Updated model coming soon!) | 8e2d3930145dc43d3084396f49d38a3f | | 8x22 Instruct | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-Instruct-v0.3.tar | 471a02a6902706a2f1e44a693813855b | | 7B Base | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-v0.3.tar | 0663b293810d7571dad25dae2f2a5806 | | 8x7B | Updated model coming soon! | - | | 8x22B | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-v0.3.tar | a2fa75117174f87d1197e3a4eb50371a | | Codestral 22B | https://models.mistralcdn.com/codestral-22b-v0-1/codestral-22B-v0.1.tar | 1ea95d474a1d374b1d1b20a8e0159de3 | | Mathstral 7B | https://models.mistralcdn.com/mathstral-7b-v0-1/mathstral-7B-v0.1.tar | 5f05443e94489c261462794b1016f10b | | Codestral-Mamba 7B | https://models.mistralcdn.com/codestral-mamba-7b-v0-1/codestral-mamba-7B-v0.1.tar | d3993e4024d1395910c55db0d11db163 | | Nemo Base | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-base-2407.tar | c5d079ac4b55fc1ae35f51f0a3c0eb83 | | Nemo Instruct | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar | 296fbdf911cb88e6f0be74cd04827fe7 | | Mistral Large 2 | https://models.mistralcdn.com/mistral-large-2407/mistral-large-instruct-2407.tar | fc602155f9e39151fba81fcaab2fa7c4 |

Note:

• Important:
• mixtral-8x22B-Instruct-v0.3.tar is exactly the same as Mixtral-8x22B-Instruct-v0.1, only stored in .safetensors format
• mixtral-8x22B-v0.3.tar is the same as Mixtral-8x22B-v0.1, but has an extended vocabulary of 32768 tokens.
• codestral-22B-v0.1.tar has a custom non-commercial license, called Mistral AI Non-Production (MNPL) License
• mistral-large-instruct-2407.tar has a custom non-commercial license, called Mistral AI Research (MRL) License
• All of the listed models above support function calling. For example, Mistral 7B Base/Instruct v3 is a minor update to Mistral 7B Base/Instruct v2, with the addition of function calling capabilities.
• The "coming soon" models will include function calling as well.
• You can download the previous versions of our models from our docs.

From Hugging Face Hub

| Name | ID | URL | |-------------|-------|-------| | Pixtral Large Instruct | mistralai/Pixtral-Large-Instruct-2411 | https://huggingface.co/mistralai/Pixtral-Large-Instruct-2411 | | Pixtral 12B Base | mistralai/Pixtral-12B-Base-2409 | https://huggingface.co/mistralai/Pixtral-12B-Base-2409 | | Pixtral 12B | mistralai/Pixtral-12B-2409 | https://huggingface.co/mistralai/Pixtral-12B-2409 | | Mistral Small 3.1 24B Base | mistralai/Mistral-Small-3.1-24B-Base-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503 | Mistral Small 3.1 24B Instruct | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503 |

Usage

News!!!: Mistral Large 2 is out. Read more about its capabilities here.

Create a local folder to store models

export MISTRAL_MODEL=$HOME/mistral_models
mkdir -p $MISTRAL_MODEL

Download any of the above links and extract the content, *e.g.*:

export 12B_DIR=$MISTRAL_MODEL/12B_Nemo
wget https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar
mkdir -p $12B_DIR
tar -xf mistral-nemo-instruct-2407.tar -C $12B_DIR

or

export M8x7B_DIR=$MISTRAL_MODEL/8x7b_instruct
wget https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar
mkdir -p $M8x7B_DIR
tar -xf Mixtral-8x7B-v0.1-Instruct.tar -C $M8x7B_DIR

For Hugging Face models' weights, here is an example to download Mistral Small 3.1 24B Instruct:

from pathlib import Path
from huggingface_hub import snapshot_download

mistral_models_path =…