Fireattention V3

FireAttention V3: Enabling AMD as a viable alternative for GPU inference

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Fireattention V3 FireAttention V3: Enabling AMD as a viable alternative for GPU inference

PUBLISHED 10/15/2024

Table of Contents FireAttention V3 Porting To AMD Learnings

Hardware

Software

FireAttention V3 Benchmarks Metrics

LLaMA 70B

LLaMA 8B Conclusions

Table of Contents

This post is the continuation of our FireAttention blog series: FireAttention V1 and FireAttention V2 . This time we are going to focus on a different GPU hardware, namely AMD MI300 GPU . While spec-wise it looks quite superior to NVIDIA H100 GPU we never know how it’s going to perform in real-world LLM inference settings until we run benchmarks, which represent practical LLM usage. Fireworks has been using AMD MI300 hardware in production since the launch of LLaMA 405b . In this post we are going to go over the work which made it happen. FireAttention V3

FireAttention V3 is an AMD-specific implementation for Fireworks LLM. When measured on 8 MI300 GPUs vs other leading LLM implementations (NIM Containers on H100 and AMD vLLM on MI300) it achieves 1.4x improvement for the average RPS @ 8 secs metric for LLaMA 8B model and 1.8x improvement for the average RPS @ 10 secs for LLaMA 70B model. In some low-latency scenarios RPS improvement can reach up to ~3x for NIM and up to ~5.5x for AMD vLLM . It even improves minimal achievable latency for LLaMA 8B model by 1.6x .

Porting To AMD Learnings

To tell the truth, we were pleasantly surprised at how smooth it was to port our proprietary stack to ROCm and get to functional parity - i.e. all functionality working correctly. We leveraged PyTorch’s official ROCm support , which matured a lot over the past few years. For custom CUDA code, with the exception of advanced APIs, most of API calls are supported on ROCm and are converted automatically when running the hipify utility. That said, automatic porting falls short when the best performance is needed. Although there are few HIP porting guides , we’ve decided to emphasize some LLM-specific performance gotchas, some of which are not widely covered. Also, most of the HIP porting guides are written by AMD themselves, while AMD is not incentivized to cover AMD performance issues, while we being a neutral third-party can do a more unbiased analysis. Hardware

Warp size is 64 on AMD vs 32 on NVIDIA. There are still 4 SIMDs (same as 4 schedulers on NVIDIA SMs), and there are still 32 shared memory banks, each 32 bits wide (this makes bank conflicts a bit trickier to reason about). Number of SMs (aka Compute Units) is significantly larger: 304 (MI300) vs 113 (H100), but shared memory is way smaller: 64 KB (shared memory) + 32 KB (L1) on MI300 vs 256 combined shared memory + L1 on H100. HBM is much larger: 192 GBs vs 80 GBs. We find theoretical FLOPs and memory bandwidth numbers to be almost useless . They provide only the upper limit, while achievable numbers can be practically very different. Thus it’s extremely important to benchmark the actual operations. Flops-heavy gemms are inferior to NVIDIA cuBLAS / CUTLASS . It’s also reported by other parties for BF16 . We see a similar difference for FP8: ~1.2 PFLOPs MI300 vs ~1.5 PFLOPs H100. We attribute this inferiority to the suboptimal power management based on the new Smart Shift technology . We hope that it can be improved in the upcoming firmware upgrades. Curious readers can just compare performance of gemms with real data vs just zeros. On MI300, this difference reaches up to 2x, while much less on H100. Gritty details are here . Memory bandwidth is higher on MI300 . Although we can’t quite reach advertised numbers (likely due to the fact that we use NPS 1 partitioning and there are 2 HBM banks per XSD), achieved memory bandwidth on MI300 is still better vs H100. Software

bfloat16 types: __nv_bfloat16 vs __hip_bfloat16 (and other vector types), float8 types: __nv_fp8_e4m3 vs __hip_fp8_e4m3_fnuz (and other vector types) are not covered by the hipify utility and have to be taken care by a programmer. Although cooperative groups API is partially supported, the most important reductions operations are not. Thus it’s better not to use cooperative groups API for portable code and prefer more narrowly targeted portable libraries like NVIDIA CUB vs hipCUB Note, these types are different: NVIDIA FP8 vs AMD FP8 . This practically means that max is 448 (nvidia) vs 240 (amd). At the same time we found that AMD fp8 numerics typically yield a bit better accuracy according to our quantization evaluation benchmarks (although this difference is quite negligible compared to other differences, like whether to use FP8 for QKV matrix, whether to use channel scaling etc). hipBLASLt is the ‘state-of-the-art’ library for matrix multiplications. It supports both fp16 and fp8 compute (including mixed precision). Although grouped gemms are supported, fp8 currently only works over mixed precision (hence only fp16 compute). This makes MoE models currently non-competitive on AMD . We are looking forward to hipBLASLt library to support fp8 compute for grouped gemms. FireAttention V3

Kernel-level LLMs optimization is mostly about two operations: matmuls and attention (which in turn is a specialized fused matmul). To achieve the best performance, we had to re-write our attention kernel from scratch. The reason is that performant versions of attention have to rely on matrix core ops (tensor core on NVIDIA), while shapes and element swizzling formats are totally different on AMD. Although there is async copy support on AMD, there is no TMA yet. Combined with quite different shared memory size, all these differences lead to totally different design choices. Benchmarks

Fireworks excels at providing cost-effective low-latency solutions for high traffic use cases. We picked two quite common cases both demanding low latencies while requiring to handle high volume of requests per second (RPS). We are running all models in fp8 precision (weights, activations and kv-caches) to ensure that we fully utilize GPU compute capabilities. There are many LLM implementations for NVIDIA hardware. To keep the highest baseline we chose to compare against NIM Containers , which is based on TensorRT LLM , one of the most performant OSS libraries. Unlike TensorRT LLM, which is notoriously hard to build and run, NIM is a breeze to...