google-deepmind/scivid

google-deepmind/scivid

Language: Python

License: Apache-2.0

Stars: 17

Forks: 4

Open issues: 1

Created: 2025-07-04T15:41:20Z

Pushed: 2026-03-02T14:41:57Z

Default branch: main

Fork: no

Archived: no

README:

SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications

Welcome to the official Google DeepMind repository for SciVid, a comprehensive benchmark for evaluating Video Foundation Models (ViFMs) across multiple scientific disciplines.

This repository contains:

• Instructions to download the datasets in a format compatible with our

evaluations

• Instructions to evaluate pretrained video models on SciVid (eg. Hugging Face VideoMAE-B backbone), easily extensible to evaluate your own.
• [![Open In

Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google-deepmind/scivid/blob/main/colabs/demo.ipynb) demo to inspect SciVid benchmark data, models and predictions.

Overview

SciVid comprises five Scientific Video tasks, combining both established and under-explored tasks across medical computer vision, animal behavior understanding and weather forecasting, and covering diverse data distributions and training regimes.

| Dataset | FlyVsFly | CalMS21 | WeatherBench 2 | Digital Typhoon | STIR | |-------------------|-------------------------------|-------------------------------|---------------------------------|---------------------------------|--------------------------------| | Example |  |  |  |  |  | | Domain | Fly behaviour | Mice behaviour | Weather | Typhoon satellite images | Surgical tissue | | Task | Classification | Classification | Forecasting | Central pressure forecasting | Point tracking | | Num train samples | 1M | 27K | 57K | 696 | N/A |

During evaluation, each model is appended with task-specific readouts and fine-tuned on each downstream dataset, with or without freezing the backbone.

 *Evaluation overview. For each task, we train a lightweight readout on top of the backbone, which is frozen (❄️) or fine-tuned (🔥).*

In our paper, we evaluate the capabilities of a large set of ViFMs, establishing strong baselines and demonstrating the potential for effective transfer learning.

Installation

Installation has been tested with cuda 12.4 and python 3.10.

Get the code from the GitHub repository

git clone git@github.com:google-deepmind/scivid.git

Create and activate scivid conda environment

After installing miniconda if needed, create a conda environment with all required scivid dependencies.

conda env create -f scivid/environment.yml

This will create a conda environment named scivid, which you can activate with

conda activate scivid

Setup data

Download data

For optimized training, download the data from the scivid cloud storage bucket. We also recommend storing the data on a local SSD drive, if you have one available. This is particularly important to speed up training on the weatherbench_future_pred task.

export SCIVID_COPY=/path/to/scivid_data_copy # set to the desired path (on ssd if available)
mkdir -p $SCIVID_COPY
gcloud storage rsync --recursive gs://scivid $SCIVID_COPY

Alternatively (slower), you can mount the data using gcsfuse in a separate location by running:

export SCIVID_MOUNT=/path/to/scivid_data_mount # set to *a separate location* from SCIVID_COPY
mkdir -p $SCIVID_MOUNT
gcsfuse --implicit-dirs scivid $SCIVID_MOUNT

In this case, we still recommend downloading the data for the weatherbench_future_pred task with:

mkdir -p $SCIVID_COPY/full/weatherbench
gcloud storage rsync --recursive gs://scivid/full/weatherbench $SCIVID_COPY/full/weatherbench

Usage

Manage accelerator visibility and resources

To define which GPU to use and properly manage the accelerator memory, you will need to set the following environment variables:

export CUDA_VISIBLE_DEVICES=0
export XLA_PYTHON_CLIENT_MEM_FRACTION=.5
export TF_GPU_ALLOCATOR=cuda_malloc_async

What these commands do:

CUDA_VISIBLE_DEVICES=0 ensures data workers have access to the accelerator when needed.

XLA_PYTHON_CLIENT_MEM_FRACTION=.5 reduces jax GPU memory pre-allocation, ensuring enough GPU memory is available for other processes.

TF_GPU_ALLOCATOR=cuda_malloc_async helps prevent out-of-memory errors by avoiding memory fragmentation issues.

Increase maximum number of open files

You might need to increase the maximum number of files which can be simultaneously opened to enable parallelized data preprocessing.

ulimit -n 4096

Run training

Set the SCIVID_DATA_DIR environment variable to either the root of the copied or mounted data, depending on which data source you intend to use.

export SCIVID_DATA_DIR=$SCIVID_COPY # or $SCIVID_MOUNT

Below, we provide an example training command for training the task-specific readout using frozen features from the VideoMAE-B backbone on the Fly vs. Fly task (on GPU).

python -m kauldron.main --cfg=scivid/configs/launch_config.py:hf_videomae:flyvsfly_classification --cfg.workdir=/home/${USER}/tmp/exps/flyvsfly_videomae --cfg.aux.platform='cuda' --pdb

For WeatherBench2 forecasting, we additionally set XLA_FLAGS="--xla_gpu_autotune_level=0" to avoid memory errors as follows:

XLA_FLAGS="--xla_gpu_autotune_level=0" python -m kauldron.main…