nvidia/NV-JEPA-DNA-NTv3

Model Overview

JEPA-DNA-NTv3-100M is the Nucleotide Transformer v3 (NTv3) 100M backbone of the JEPA-DNA family, a model-agnostic continual pre-training framework that grounds genomic foundation models by coupling standard token-level DNA language modeling with a Joint-Embedding Predictive Architecture (JEPA). Where conventional genomic models rely purely on masked language modeling and prioritize local token reconstruction, JEPA-DNA additionally supervises the model's global sequence embedding in a latent space, forcing it to predict the functional representation of masked genomic segments rather than their literal tokens. This shifts the learning signal from token recovery toward semantic alignment, yielding more linearly-separable and biologically relevant features while preserving the generative precision of the underlying backbone. This checkpoint is an exact parameter match to the original research work described in the JEPA-DNA paper, and the code for using it can be found in the official JEPA-DNA Github repository.

Description:

JEPA-DNA-NTv3-100M is the NTv3 variant of JEPA-DNA, a genomic foundation model training framework that adds a Joint-Embedding Predictive Architecture (JEPA) objective to standard DNA language-model pretraining. It applies a continual pre-training stage on top of the NTv3 100M backbone — a 100M-parameter hybrid U-Net / Transformer model trained with a masked language modeling (MLM) objective over single nucleotides — combining token-level learning with latent-space prediction to improve functional genomic representations.

This model is ready for non-commercial use.

License/Terms of Use:

Governing Terms: Use of this model is governed by the NVIDIA non-commercial license, subject to completion of NVIDIA's applicable internal legal and open source release review process.

Deployment Geography:

Global

Use Case:

JEPA-DNA checkpoints are intended for genomics research, representation learning, and downstream sequence modeling workflows. Typical uses include feature extraction, linear probing, continual pre-training studies, and zero-shot embedding-based scoring of sequence perturbations. This model is not intended to be used as diagnostic tools, clinical decision systems, or validated medical products.

Release Date:

Github 05/18/2026 via https://github.com/NVIDIA-Digital-Bio/JEPA-DNA

Hugging Face 07/16/2026 via https://huggingface.co/nvidia/NV-JEPA-DNA-NTv3-100M

References(s):

• JEPA-DNA paper

• JEPA-DNA GitHub repository

Model Architecture:

JEPA-DNA-NTv3-100M uses NTv3 100M, a 100M-parameter hybrid U-Net / Transformer model trained with a masked language modeling (MLM) objective over single nucleotides, with sequence aggregation based on mean pooling over token embeddings. JEPA-DNA augments this backbone with a target encoder (an exponential-moving-average copy of the backbone) and a lightweight 3-layer MLP predictor that operates directly on the pooled sequence embedding to reconstruct the target's latent representation. Training uses span-based masking and re-masking to prevent trivial recovery of masked targets, and proceeds in two stages: first optimizing the predictor with a frozen backbone, then joint optimization of predictor and backbone under a composite objective combining the native MLM loss, a JEPA cosine-similarity latent-prediction loss, and VICReg-style variance and covariance regularization:

$$\min_{\theta,\phi}\; \mathcal{L}_{total} = \lambda_1 \mathcal{L}_{llm} + \lambda_2 \mathcal{L}_{jepa} + \lambda_3 \mathcal{L}_{var} + \lambda_4 \mathcal{L}_{cov}$$

| Model Name | Parameters | |-------------|-------------:| | JEPA-DNA-NTv3-100M | 1.00 × 10⁸ |

Input:

Input Type(s): Text (genomic DNA sequences, tokenized)

Input Format: DNA String

Input Parameters: 1D

Other Properties Related to Input: Input DNA sequences context length are up to 8,192 bp, single-nucleotide tokenization, and preprocessing, with sequence aggregation based on mean pooling over token embeddings.

Output:

Output Type(s): Embeddings / latent sequence representations

Output Format: Embeddings (2D, sequence-length × hidden-dim) and sequence representations (1D, hidden-dim)

Output Parameters: 2D (embeddings), 1D (sequence representation)

Other Properties Related to Output: Outputs are sequence-level embeddings and hidden states that can be used for linear probing, embedding-based scoring of sequence perturbations, and retrieval-style tasks; they do not directly produce labels but are intended for downstream models or evaluators.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s):

• PyTorch - 2.6.0

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell

• NVIDIA A10G

Preferred/Supported Operating System(s):

• Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

• JEPA-DNA-NTv3-100M-v1

Training, Testing, and Evaluation Datasets:

Training Dataset:

Properties: Continual pre-training uses the OpenGenome2 corpus (random crops of 8,192 bp), approximately 384M records. Content is genomic DNA sequences encoded as text-like sequences of single-nucleotide tokens. By maintaining data parity with the original baseline, the continual-training stage isolates the impact of the JEPA-DNA objective.

Data Modality

• Text (genomic DNA sequences)

Training Data Size: Approximately 384M genomic records.

Data Collection Method by dataset:

• Hybrid: Automated, Automatic/Sensors

**Labeling...