Snowflake/Arctic-LSTM-Speculator-gpt-oss-120b

ArcticSpeculator

Build the fastest OSS vllm-based speculative decoding system for your own model, using ArcticTraining and ArcticInference!

Throughput (tokens/s) of gpt-oss-120b on 8xH100 using vLLM below:

| method | ShareGPT | HumanEval | |--------------------------------------|----------------|--------------| | vLLM V1 Baseline | 220.2 | 220.7 | | ArcticSpeculator | 377.3 | 400.0 |

For more details about ArcticSpeculator and how to use it:

• ❄️ [Using Arctic-Inference and Arctic-Training for improving real-world speculative decoding Performance (blog)]()
• 🚀 Getting started guide using ArcticTraining

See all of the speculators we have released via our Speculators Collection