NVIDIA/Audio2Face-3D

NVIDIA/Audio2Face-3D

Description: repo collection for NVIDIA Audio2Face-3D models and tools

Stars: 444

Forks: 52

Open issues: 2

Created: 2025-09-16T19:44:26Z

Pushed: 2025-09-24T23:18:36Z

Default branch: release

Fork: no

Archived: no

README:

NVIDIA Audio2Face-3D

 Audio2Face-3D is an advanced technology that generates high-fidelity 3D facial animation from an audio source, supporting both pre-recorded files and real-time streams.

The system analyzes vocal data to synthesize detailed and realistic articulation. This includes the precise, synchronized motion of the jaw, tongue, and eyes, as well as the subtle deformations of the facial skin. This comprehensive approach results in two key outputs:

• Accurate lip-sync based on phonetic analysis.
• Nuanced emotional expression inferred from the tone of the speech.

Together, these elements produce a complete and lifelike facial performance that matches the source audio. Audio2Face-3D drives a character's facial performance through direct mesh deformations, joint transformations, or blend shape weights.

This page serves as the central hub for all official Audio2Face-3D technologies and tools, including pre-trained models, the development SDK, plugins for Autodesk Maya and Unreal Engine 5, as well as sample datasets.

Audio2Face-3D Collection

The Audio2Face-3D collection is distributed across several platforms. The collection offers the following components:

• NVIDIA Repositories: Access source code, helper scripts, packaged builds, and documentation.

• Hugging Face Hub: Download pre-trained models and sample training datasets.

NVIDIA Repositories

Package Use Format License Info

Audio2Face-3D SDK Authoring and runtime facial animations on-device or in the cloud. Source Code - C++ & Python MIT More

Audio2Face-3D Training Framework Framework (v1.0) for creating Audio2Face-3D models with your data. Source Code - Python & Docker Container Apache More

Maya ACE (MACE) Animation authoring plugin with local execution (v2.0). Autodesk Maya Plugin Module, Source Code C++, and Python MIT More

Unreal Engine 5 plugin Unreal Engine 5 plugin (v2.5) for UE 5.5 and UE 5.6 game engine workflows. Unreal Engine (.uplugin) & Source Code C++ MIT More

Audio2Face-3D NIM Simplifies the large-scale deployment of Audio2Face-3D models. Docker Container NVIDIA Software License Agreement and Product Specific Terms for AI Products More

Hugging Face Hub

Package Use Format License Links

Audio2Face-3D Models Regression (v2.3) and diffusion (v3.0) pre-trained models for generating lip-sync. Open Weights / onnx-trt Nvidia Open Model v3.0 Mark v2.3 Claire v2.3.1 James v2.3.1

Audio2Emotion Models Production (v2.2) and experimental (v3.0) models to infer emotional state from audio. Open Weights / onnx-trt Custom (use allowed with Audio2Face only) v2.2 v3.0

Audio2Face-3D Training Sample Data Example dataset to get started with the training framework. Audio files, blendshape data, animated geometry caches, geometry files, and transform files. Custom (evaluation only) Claire Dataset v1.0.0

Audio2Face-3D SDK

Target audience: C++ developers, tools & pipeline engineers

Link: A2X SDK on Github

The Audio2Face-3D SDK is a powerful C++ library designed for developers who need to integrate Audio2Face-3D's real-time inference capabilities directly into their own software. It provides the necessary tools to build custom applications, plugins, or batch processing tools that can generate high-quality facial animation data from an audio source using any compatible Audio2Face-3D model.

Key Features

• Cross-Platform C++ Library: Ensures wide compatibility across different operating systems.
• Real-Time Performance: Highly optimized for low-latency inference suitable for interactive applications.
• Hardware Acceleration: Leverages NVIDIA GPUs for maximum performance, with a CPU fallback for broader hardware support.

Best Use Cases

• Game Development: Integrate real-time facial animation for in-game character dialogue and cinematics.
• Custom Applications: Add Audio2Face-3D functionality to proprietary content creation tools.
• Production Deployment: Build scalable, automated facial animation pipelines for games, films, or broadcast.

Audio2Face-3D Training Framework

Target audience: 3D animators, animation technical and art directors, Python scripters, ML researchers

Link: Audio2Face-3D Training Framework on Github

The Audio2Face-3D Training Framework provides the complete toolset required to create custom, high-performance facial animation models from your own datasets. This is the same powerful framework used by NVIDIA to train the official Mark, Claire, and James pre-trained models.

By training on your own character's lip-synced animations, you can generate models that perfectly capture a desired performance style, personality, or language. A provided sample dataset helps users understand the complete workflow from data preparation to a fully trained model.

Key Features

• Custom Model Training: Train unique Audio2Face-3D models from scratch on your proprietary data.
• Flexible Environment: A Python-based framework distributed via Docker for consistent and reproducible setups.
• Multi-Language Support: Natively supports training on custom datasets in single or multiple languages.
• Standardized Output: Exports trained models along with a JSON model card for seamless integration with other Audio2Face tools.

Best Use Cases

• Develop unique models that capture a character's specific personality and performance style.
• Create language-specific models to support multilingual dialogue and localization.
• Build highly specialized models to meet unique research, pipeline, or production requirements.

Maya ACE Plugin (MACE)


Target audience: 3D animators, animation technical directors, Python scripters

Link: Maya ACE on Github

MACE is a comprehensive plugin that integrates the full power of Audio2Face-3D directly within the Autodesk Maya environment.

It provides an intuitive user interface for generating real-time facial animation from an audio source, with the flexibility to output to direct mesh deformations, object transformations,...