NVIDIA/Nvidia-Comms-Perf-Suite

NVIDIA/Nvidia-Comms-Perf-Suite

Description: A comprehensive toolkit for GPU Communications Libraries performance testing and data analysis.

Language: Python

License: NOASSERTION

Stars: 9

Forks: 5

Open issues: 2

Created: 2025-09-18T16:52:22Z

Pushed: 2026-01-06T13:05:21Z

Default branch: main

Fork: no

Archived: no

README:

Nvidia Comms Perf Suite

A comprehensive toolkit for GPU Communications Libraries performance testing and data analysis.

Overview

Nvidia Comms Perf Suite is a collection of utilities designed to streamline the process of testing and analyzing performance data from NVIDIA GPU Communications Libraries. The toolkit provides:

1. Test Script Generation: Generate test scripts for performance benchmarking across different systems and configurations. 2. Result Processing: Process test results using one of three approaches:

• Convert: Save results to local files in various formats (e.g., CSV) for analysis or sharing
• Upload: Store results directly in databases (PostgreSQL, MySQL, SQLite) and/or cloud storage (S3)
• Report: Generate comprehensive PDF performance analysis reports with customizable charts and visualizations

Installation

Virtual Environment (Recommended)

We recommend using a virtual environment to install Nvidia Comms Perf Suite. This helps isolate dependencies and avoid conflicts with other Python packages:

# Create a virtual environment
python -m venv nvcomms-perf-env

# Activate the virtual environment
source nvcomms-perf-env/bin/activate

Build the package with uv

We use uv to build distributable artifacts (sdist/wheel):

# From the repo root
uv build

# Artifacts will be created under dist/
ls -l dist/

PostgreSQL URL Format

Nvidia Comms Perf Suite uses psycopg3 by default. Use this URL format:

• PostgreSQL: postgresql+psycopg://user:password@host:port/database

Quick Start

CLI usage

> nvcomms-perf --help

Usage: nvcomms-perf [OPTIONS] COMMAND [ARGS]...

Performance Tools for NVIDIA GPU Communications Libraries

Options:
-v, --verbose Increase verbosity (-v, -vv)
--version Show version and exit
--install-completion Install completion for the current shell.
--show-completion Show completion for the current shell, to copy it or
customize the installation.
--help Show this message and exit.

Commands:
generate-job-script Generate job scripts for GPU Comm Libraries...
parse-and-upload (Deprecated) Use [bold red]convert[/bold red] instead
convert Convert test results to a local file in the...
upload Upload test results to a database (PostgreSQL,...
report Generate performance analysis reports from test results

Workflow

The typical workflow for using nvcomms-perf is:

1. Generate test scripts using generate-job-script 2. Run the tests on your cluster (produces test output files) 3. Process results using one of these options:

• convert: Save results to a local file (CSV, etc.)
• upload: Upload results directly to a database and/or S3
• report: Create PDF report from test tarball or CSV

Step 1: Test Generation

Generate job scripts for performance testing:

# Generate a job script and print to stdout
nvcomms-perf generate-job-script --system CLUSTER_1 --systems-toml systems.toml --user-toml user.toml --testset-toml testset.toml

# Generate a job script and save to file
nvcomms-perf generate-job-script --system CLUSTER_1 --systems-toml systems.toml --user-toml user.toml --testset-toml testset.toml --output job.sbatch

# Generate with test report
nvcomms-perf generate-job-script --system CLUSTER_1 --systems-toml systems.toml --user-toml user.toml --testset-toml testset.toml --generate-test-report junit --output job.sbatch

Step 2: Run Tests

After generating the job script, submit it to your cluster:

# Submit the job
sbatch job.sbatch

# The job will produce test output files (logs, tarballs, etc.)

Step 3: Process Results

Choose one of the following options to process your test results:

Option A: Convert to Local File

Use convert to save results to a local file for analysis, sharing, or archival:

# Convert a single NCCL test output to a local CSV file
nvcomms-perf convert -i nccl-tests:/path/to/nccl-tests.log -o results.csv

# Convert test results from multiple tarballs to a single local CSV file
nvcomms-perf convert -i tarball:/path/to/archive1.tar.gz -i tarball:/path/to/archive2.tar.gz -o combined_results.csv

# Convert with explicit output format (CSV) and save to local file
nvcomms-perf convert -i nccl-tests:/path/to/test.log -o csv:/tmp/results.csv

# Convert tarball to local CSV file without progress bar
nvcomms-perf convert -i tarball:/path/to/archive.tar.gz -o results.csv --hide-progress

# Convert and stop on first error (instead of continuing)
nvcomms-perf convert -i tarball:/path/to/archive.tar.gz -o results.csv --stop-on-error

# Convert using custom number of parallel workers for faster processing
nvcomms-perf convert -i tarball:/path/to/archive.tar.gz -o results.csv --max-workers 8

Option B: Upload to Database/S3

Use upload to directly store results in a database and/or S3:

# Upload to PostgreSQL database
nvcomms-perf upload -i tarball:/path/to/archive.tar.gz --db-url postgresql+psycopg://user:pass@host:port/database

# Upload to S3
nvcomms-perf upload -i tarball:/path/to/archive.tar.gz --s3-bucket my-bucket --s3-region us-east-1

# Upload to both PostgreSQL and S3
nvcomms-perf upload -i tarball:/path/to/archive.tar.gz --db-url postgresql+psycopg://user:pass@host:port/database --s3-bucket my-bucket

# Upload multiple files
nvcomms-perf upload -i tarball:/path/to/archive1.tar.gz -i tarball:/path/to/archive2.tar.gz --db-url postgresql+psycopg://user:pass@host:port/database

# Use environment variables for credentials
export NVCOMMS_PERF_DB_URL="postgresql+psycopg://user:pass@host:port/database"
export NVCOMMS_PERF_S3_BUCKET="my-bucket"
export NVCOMMS_PERF_S3_REGION="us-east-1"
nvcomms-perf upload -i tarball:/path/to/archive.tar.gz

Option C: Generate Performance Reports

Use report to create comprehensive PDF performance analysis reports with various chart types:

Available PDF Report Types:

• Basic Report (basic_report):

Basic report for NCCL includes:

• Default Configuration Comparison: Curves run with default NCCL algorithm and protocol. One NCCL_TESTS_SPLIT_MASK and scale (num nodes and num processes)...