NVIDIA/metropolis-sdg-smart-cities

NVIDIA/metropolis-sdg-smart-cities

Description: Metropolis Smart-City Synthetic Data Generation Recipe allows you to generate synthetic data and then augment them. The generated data then can be used to fine-tune VLM models such as Cosmos Reason (CR).

Language: Python

License: Apache-2.0

Stars: 16

Forks: 4

Open issues: 0

Created: 2025-11-11T01:48:18Z

Pushed: 2025-12-01T20:15:24Z

Default branch: main

Fork: no

Archived: no

README:

Synthetic Data Generation for Smart City Applications

An end-to-end workflow for generating photo-realistic synthetic data for Vision-Language Models (VLMs) training on traffic scenarios. The pipeline simulates scenes in CARLA, extracts rich ground truth, applies COSMOS photo-realistic augmentation, and performs SoM-aligned post-processing.

• What it does: Turns CARLA simulation recordings into curated, photo-realistic videos and annotations for VLM training.
• Who it’s for: Machine Learning and Deep Learning engineers and researchers building perception and VLM systems for Smart City applications.
• How it runs: Docker Compose stack with a Workbench, CARLA server, NVIDIA VLM/LLM models, and NVIDIA Cosmos-Transfer.

 ---

Contents

• [Synthetic Data Generation for Smart City Applications](#synthetic-data-generation-for-smart-city-applications)
• [Overview](#overview)
• [Architecture](#architecture)
• [Prerequisites](#prerequisites)
• [Obtain API keys](#obtain-api-keys)
• [Workflow Inputs](#workflow-inputs)
• [Maps](#maps)
• [Scenario Logs](#scenario-logs)
• [Recorder Config](#recorder-config)
• [System Requirements](#system-requirements)
• [Quickstart (Docker Compose)](#quickstart-docker-compose)
• [Advanced Configuration](#advanced-configuration)
• [Workflow Usage (Stage-by-Stage)](#workflow-usage-stage-by-stage)
• [Troubleshooting / FAQ](#troubleshooting--faq)
• [Security and Compliance](#security-and-compliance)
• [Contributing and Support](#contributing-and-support)
• [License and Citation](#license-and-citation)

---

Overview

This workflow provides a Synthetic Data Generation (SDG) recipe to produce VLM-ready datasets for smart city applications. In areas where the highest model accuracy is vital, finetuning on domain specific data is essential. Synthetic data generation and augmentation offer an easy and scalable way to collect this data to your exact specifications. However, there are significant challenges associated with creating diverse, photorealistic training data from simulators that this workflow aims to address:

• Domain Gap: While simulators provide perfect ground truth and controllable scenarios, their synthetic appearance creates a substantial domain gap that limits the performance of models trained on simulator data when deployed in real-world environments.
• Scalability Constraints: Manually crafting diverse scenarios in simulators requires substantial engineering effort and computational resources, making it prohibitively expensive to scale up data diversity.
• Limited Visual Realism: Traditional simulator outputs lack the photorealistic quality needed for robust real-world model deployment, requiring additional post-processing or domain adaptation techniques.

To remedy these issues, this workflow provides a recipe to:

• Simulate customized traffic scenarios using CARLA
• Ground-truth extraction from simulation (RGB, Depth, Segmentation, Normals, 2D/3D bounding boxes, events)
• Use COSMOS-Transfer to generate photo-realistic augmentations that bridge the sim-to-real gap
• Scale synthetic data with customizable augmentation variables
• Generate post-training datasets for model fine-tuning
• SoM-aware post-processing to preserve object correspondence across modalities
• Q&A Caption generation for VLM post-training

The output of this recipe is designed to offer a simple hand-off for further fine-tuning and deployment in the metropolis stack. Refer to the Cosmos Cookbook for Fine-tuning and VSS documentation for Deployment guides.

Primary components:

• modules/carla-ground-truth-generation: CARLA ground-truth extraction and dataset tooling
• modules/augmentation: Prompting, template/prompt generation, and COSMOS execution utilities
• modules/postprocess/postprocess_for_vlm.py: SoM-aligned overlays/validations
• deploy/compose: Docker Compose assets (deploy.sh, *.yml, .env)
• docker/Dockerfile.workbench: Notebook-centric workbench image
• notebooks/carla_synthetic_data_generation.ipynb: End-to-end notebook walkthrough

---

Architecture

• Workbench: Jupyter Lab environment with Python tooling, CARLA client, augmentation runners, and notebooks
• CARLA Server (0.9.16): Simulator for generating diverse traffic scenarios (GPU-accelerated)
• NVIDIA NIM services (optional):
• Cosmos-Reason1 for captioning and scene description
• Cosmos-Transfer2.5 for photo-realistic video augmentation
• Nemotron for template/prompt generation

Key ports:

• Workbench: 8888
• Cosmos-Reason1 NIM: 8001
• Nemotron LLM NIM: 8002
• Cosmos-Transfer Gradio: 8080 (host networking; 8003 if using bridge mapping)
• CARLA: 2000 RPC, 2001 UDP, 2002 TCP

---

Prerequisites

Obtain API keys

• NGC API key
• Steps to setup HERE
• Hugging Face Token:
• Ensure your hugging face token has access to Cosmos-Transfer2.5 checkpoints
• Get a Hugging Face Access Token with Read permission
• Install Hugging Face CLI
• Login with hf auth login.
• Read and accept the NVIDIA Open Model License Agreement
• Read and accept the terms for Cosmos-Guardrail1
• Read and accept the terms for Cosmos-Transfer2.5

Workflow Inputs

The SDG workflow requires 3 unique inputs: maps, scenario logs, and sensor...