NVIDIA/nvflow

NVIDIA/nvflow

Description: Workflow orchestration framework for end-to-end synthetic data generation (SDG), training (SFT), and evaluation pipelines built on NVIDIA's NeMo ecosystem

Language: Python

Stars: 14

Forks: 2

Open issues: 0

Created: 2026-03-11T01:14:45Z

Pushed: 2026-05-18T03:21:25Z

Default branch: main

Fork: no

Archived: no

README:

NVFlow

Workflow orchestration for end-to-end ML pipelines (data generation, training, evaluation) built on the NeMo ecosystem.

NVFlow is a workflow orchestration framework for end-to-end synthetic data generation (SDG), training (SFT), and evaluation pipelines built on NVIDIA's NeMo ecosystem. It exists to standardize how teams build, reproduce, and scale complex ML pipelines across domains with reusable stages and declarative workflows that run locally or on Slurm clusters.

It provides a structured way to build, manage, and execute pipelines through:

• Recipes - Domain-specific workflows (e.g., finance)
• Stages - Reusable pipeline components for SDG, training, and evaluation
• Workflows - YAML-based configurations that chain stages together

Key features:

• Reusability and reproducibility with a structured, stage-based architecture
• Flexible execution via CLI (nflow), Python scripts, or programmatic API
• Cluster integration with native Slurm support
• Built on NeMo leveraging NeMo-Skills, NeMo-RL, and NeMo-Gym infrastructure

Example use case: The finance recipe demonstrates a complete pipeline: download SEC filings → generate synthetic Q&A data → fine-tune models → evaluate performance, producing 300K+ synthetic Q&A pairs.

🔑 Core Concepts

Understanding the terminology is key to working effectively with NVFlow:

• Recipe: A domain-specific collection of stages and workflows organized around a particular use case (e.g., finance). Recipes provide ready-to-use pipelines for specific problem domains.

• Workflow: A declarative pipeline defined in YAML that orchestrates multiple stages in a specific order. Workflows define stage dependencies, data flow between stages, and execution configuration. Think of a workflow as a recipe that connects stages together to accomplish an end-to-end objective.

• Stage: A self-contained, reusable component that performs a single, specific task in your ML pipeline. Each stage is an independent unit of work (e.g., downloading data, generating synthetic examples, training a model). Stages are implemented as Python classes and can be composed together.

Example hierarchy:

Recipe: finance
├── Workflow: download_sec_filings (defined in YAML)
│ └── Stage: demo (or sap-500)
└── Workflow: template_based_sdg (defined in YAML)
├── Stage: create_seed_data
├── Stage: generate_questions
├── Stage: map_questions_to_context
├── Stage: generate_answers
├── Stage: genselect_answers
└── Stage: filter_answers

In practice, you define workflows in YAML configuration files, reference stages by their short names, and run them via CLI or Python API. This separation allows you to reuse stages across different workflows and maintain clean, modular pipeline code.

📁 Understanding the Folder Structure

How concepts map to folders:

nvflow/recipes/{recipe_name}/
├── workflows/*.yaml # Workflow definitions (e.g., template_based_sdg.yaml)
├── stages/{category}/*.py # Stage implementations (e.g., stages/sdg/generate_answers.py)
└── prompts/ # Prompt templates used by stages

Finding stage implementations:

When a workflow YAML references a stage like generate_answers, the Python implementation is located at:

nvflow/recipes/{recipe}/stages/{category}/{stage_name}.py

For example:

• Stage name in YAML: generate_answers
• Implementation file: nvflow/recipes/finance/stages/sdg/generate_answers.py

About category folders:

The {category} folders (like sdg/, data/, training/) are organizational containers that group related stages together. They help keep the codebase organized but do not affect stage naming - stages are always referenced by their short name in workflow YAML files, not by their folder path.

Example:

# In workflow YAML
pipeline_stages:
- download_sec_filings # Short name
- generate_questions # Short name

# These map to Python files:
# stages/download/download_sec_filings.py
# stages/sdg/generate_questions.py

📋 Prerequisites

• Git - to clone the repository
• uv - Python package manager (docs)

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

📦 Installation

git clone https://github.com/NVIDIA/nvflow.git
cd nvflow

# For users
uv sync

# For developers
uv sync --all-extras
uv run pre-commit install

> ⚠️ Developers: Always run uv run pre-commit install after cloning. This enables automatic code quality checks on every commit.

Activating the Virtual Environment (Optional)

By default, use uv run to run commands in the project's virtual environment. If you prefer to activate the environment directly:

source .venv/bin/activate

# Now you can run commands without 'uv run' prefix
python --version # 3.12+
nflow --help
pytest

🔧 Cluster Setup

To run workflows on a Slurm cluster you need to: (1) build the four NVFlow container images from the Dockerfiles in [dockerfiles/](dockerfiles/), (2) convert them to .sqsh for Slurm, and (3) write a cluster config (cluster_configs/my_cluster.yaml). The containers are self-sufficient — all dependencies are pre-installed, so no runtime downloads are needed.

> See [INSTALL.md](INSTALL.md) for the complete setup guide > (build, sanity-check, .sqsh conversion, model staging, cluster > configuration, and verification). > > See [`dockerfiles/docker_instructions.md`](dockerfiles/docker_instructions.md) > for the build / multi-arch / sanity-check reference.

Once cluster setup is complete, set the config directory:

export NEMO_SKILLS_CONFIG_DIR=/path/to/nvflow/cluster_configs

🚀 Quick Start

CLI Invocation

# List all stages (hierarchical display)
uv run nflow list-stages

# List stages for a specific recipe
uv run nflow list-stages --recipe finance

# Get stage details (full path: recipe.workflow.stage)
uv run nflow stage-info example.sdg_simple.generate_answer

# Or with flags (if you know recipe and workflow)
uv run nflow stage-info generate_answer --recipe example --workflow…