mistralai/TensorRT-LLM

mistralai/TensorRT-LLM

Description: TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

License: Apache-2.0

Stars: 17

Forks: 7

Open issues: 0

Created: 2024-02-09T14:10:24Z

Pushed: 2024-02-12T10:36:42Z

Default branch: main

Fork: yes

Parent repository: NVIDIA/TensorRT-LLM

Archived: no

README:

TensorRT-LLM =========================== A TensorRT Toolbox for Optimized Large Language Model Inference

[Architecture](./docs/source/architecture.md) | [Results](./docs/source/performance.md) | [Examples](./examples/) | [Documentation](./docs/source/)

---

Latest News

• [2024/02/06] [🚀 Speed up inference with SOTA quantization techniques in TRT-LLM](./docs/source/blogs/quantization-in-TRT-LLM.md)
• [2024/01/30] [ New XQA-kernel provides 2.4x more Llama-70B throughput within the same latency budget](./docs/source/blogs/XQA-kernel.md)
• [2023/12/04] [Falcon-180B on a single H200 GPU with INT4 AWQ, and 6.7x faster Llama-70B over A100](./docs/source/blogs/Falcon180B-H200.md)
• [2023/11/27] SageMaker LMI now supports TensorRT-LLM - improves throughput by 60%, compared to previous version
• [2023/11/13] [H200 achieves nearly 12,000 tok/sec on Llama2-13B](./docs/source/blogs/H200launch.md)
• [2023/10/22] 🚀 RAG on Windows using TensorRT-LLM and LlamaIndex 🦙
• [2023/10/19] Getting Started Guide - [Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM, Now Publicly Available

](https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/)

• [2023/10/17] [Large Language Models up to 4x Faster on RTX With TensorRT-LLM for Windows

](https://blogs.nvidia.com/blog/2023/10/17/tensorrt-llm-windows-stable-diffusion-rtx/)

2023/11/27 - Amazon Sagemaker 2023/11/17 - Perplexity ; 2023/10/31 - Phind ; 2023/10/12 - Databricks (MosaicML) ; 2023/10/04 - Perplexity ; 2023/09/27 - CloudFlare;

Table of Contents

• [TensorRT-LLM Overview](#tensorrt-llm-overview)
• [Installation](#installation)
• [Quick Start](#quick-start)
• [Support Matrix](#support-matrix)
• [Devices](#devices)
• [Precision](#precision)
• [Key Features](#key-features)
• [Models](#models)
• [Performance](#performance)
• [Advanced Topics](#advanced-topics)
• [Quantization](#quantization)
• [In-flight Batching](#in-flight-batching)
• [Attention](#attention)
• [Graph Rewriting](#graph-rewriting)
• [Benchmark](#benchmark)
• [Troubleshooting](#troubleshooting)
• [Release notes](#release-notes)
• [Change Log](#change-log)
• [Known Issues](#known-issues)
• [Report Issues](#report-issues)

TensorRT-LLM Overview

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. It also includes a backend for integration with the NVIDIA Triton Inference Server; a production-quality system to serve LLMs. Models built with TensorRT-LLM can be executed on a wide range of configurations going from a single GPU to multiple nodes with multiple GPUs (using Tensor Parallelism and/or Pipeline Parallelism).

The Python API of TensorRT-LLM is architectured to look similar to the PyTorch API. It provides users with a [functional](./tensorrt_llm/functional.py) module containing functions like einsum, softmax, matmul or view. The [layers](./tensorrt_llm/layers) module bundles useful building blocks to assemble LLMs; like an Attention block, a MLP or the entire Transformer layer. Model-specific components, like GPTAttention or BertAttention, can be found in the [models](./tensorrt_llm/models) module.

TensorRT-LLM comes with several popular models pre-defined. They can easily be modified and extended to fit custom needs. See below for a list of supported [models](#Models).

To maximize performance and reduce memory footprint, TensorRT-LLM allows the models to be executed using different quantization modes (see [examples/gpt](./examples/gpt) for concrete examples). TensorRT-LLM supports INT4 or INT8 weights (and FP16 activations; a.k.a. INT4/INT8 weight-only) as well as a complete implementation of the SmoothQuant technique.

For a more detailed presentation of the software architecture and the key concepts used in TensorRT-LLM, we recommend you to read the following [document](./docs/source/architecture.md).

Installation

After installing the NVIDIA Container Toolkit, please run the following commands to install TensorRT-LLM for x86_64 users.

# Please use the `nvidia-docker` application, using only `docker` may cause exceptions.
# Obtain and start the basic docker image environment.
nvidia-docker run --entrypoint /bin/bash -it nvidia/cuda:12.1.0-devel-ubuntu22.04

# Install dependencies, TensorRT-LLM requires Python 3.10
apt-get update && apt-get -y install python3.10 python3-pip…