basetenlabs/triton-inference-server

basetenlabs/triton-inference-server

Description: The Triton Inference Server provides an optimized cloud and edge inferencing solution.

Language: Python

License: BSD-3-Clause

Stars: 1

Forks: 0

Open issues: 0

Created: 2024-01-09T17:50:05Z

Pushed: 2024-01-11T22:20:17Z

Default branch: main

Fork: yes

Parent repository: triton-inference-server/server

Archived: no

README:

Triton Inference Server

> [!WARNING] > ##### LATEST RELEASE > You are currently on the main branch which tracks under-development progress towards the next release. > The current release is version 2.41.0 and corresponds to the 23.12 container release on NVIDIA GPU Cloud (NGC).

Triton Inference Server is an open source inference serving software that streamlines AI inferencing. Triton enables teams to deploy any AI model from multiple deep learning and machine learning frameworks, including TensorRT, TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton Inference Server supports inference across cloud, data center, edge and embedded devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference Server delivers optimized performance for many query types, including real time, batched, ensembles and audio/video streaming. Triton inference Server is part of NVIDIA AI Enterprise, a software platform that accelerates the data science pipeline and streamlines the development and deployment of production AI.

Major features include:

• [Supports multiple deep learning

frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton)

• [Supports multiple machine learning

frameworks](https://github.com/triton-inference-server/fil_backend)

• [Concurrent model

execution](docs/user_guide/architecture.md#concurrent-model-execution)

• [Dynamic batching](docs/user_guide/model_configuration.md#dynamic-batcher)
• [Sequence batching](docs/user_guide/model_configuration.md#sequence-batcher) and

[implicit state management](docs/user_guide/architecture.md#implicit-state-management) for stateful models

• Provides Backend API that

allows adding custom backends and pre/post processing operations

• Supports writing custom backends in python, a.k.a.

Python-based backends.

• Model pipelines using

[Ensembling](docs/user_guide/architecture.md#ensemble-models) or Business Logic Scripting (BLS)

• [HTTP/REST and GRPC inference

protocols](docs/customization_guide/inference_protocols.md) based on the community developed KServe protocol

• A [C API](docs/customization_guide/inference_protocols.md#in-process-triton-server-api) and

[Java API](docs/customization_guide/inference_protocols.md#java-bindings-for-in-process-triton-server-api) allow Triton to link directly into your application for edge and other in-process use cases

• [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server

throughput, server latency, and more

New to Triton Inference Server? Make use of these tutorials to begin your Triton journey!

Join the Triton and TensorRT community and stay current on the latest product updates, bug fixes, content, best practices, and more. Need enterprise support? NVIDIA global support is available for Triton Inference Server with the NVIDIA AI Enterprise software suite.

Serve a Model in 3 Easy Steps

# Step 1: Create the example model repository
git clone -b r23.12 https://github.com/triton-inference-server/server.git
cd server/docs/examples
./fetch_models.sh

# Step 2: Launch triton from the NGC Triton container
docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:23.12-py3 tritonserver --model-repository=/models

# Step 3: Sending an Inference Request
# In a separate console, launch the image_client example from the NGC Triton SDK container
docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:23.12-py3-sdk
/workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

# Inference should return the following
Image '/workspace/images/mug.jpg':
15.346230 (504) = COFFEE MUG
13.224326 (968) = CUP
10.422965 (505) = COFFEEPOT

Please read the [QuickStart](docs/getting_started/quickstart.md) guide for additional information regarding this example. The quickstart guide also contains an example of how to launch Triton on [CPU-only systems](docs/getting_started/quickstart.md#run-on-cpu-only-system). New to Triton and wondering where to get started? Watch the Getting Started video.

Examples and Tutorials

Check out NVIDIA LaunchPad for free access to a set of hands-on labs with Triton Inference Server hosted on NVIDIA infrastructure.

Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM are located in the NVIDIA Deep Learning Examples page on GitHub. The NVIDIA Developer Zone contains additional documentation, presentations, and examples.

Documentation

Build and Deploy

The recommended way to build and use Triton Inference Server is with Docker images.

• [Install Triton Inference Server with Docker containers](docs/customization_guide/build.md#building-with-docker) (*Recommended*)
• [Install Triton Inference Server without Docker containers](docs/customization_guide/build.md#building-without-docker)
• [Build a custom Triton Inference Server Docker container](docs/customization_guide/compose.md)
• [Build Triton Inference Server from source](docs/customization_guide/build.md#building-on-unsupported-platforms)
• [Build Triton Inference Server for Windows 10](docs/customization_guide/build.md#building-for-windows-10)…