google-deepmind/thunnini

google-deepmind/thunnini

Description: Experimentation library for comparing fine-tuners for neural sequential predictors.

Language: Python

License: Apache-2.0

Stars: 10

Forks: 3

Open issues: 0

Created: 2025-05-22T19:46:23Z

Pushed: 2025-10-17T14:26:56Z

Default branch: main

Fork: no

Archived: no

README:

Thunnini

Thunnini Demo notebook: 

Full Thunnini experiment notebook: 

For more details and use-cases, see Thunnini's main publication:

Understanding Prompt Tuning and In-Context Learning via Meta-Learning (Genewein et al. 2025, NeurIPS 2025, arXiv, NeurIPS poster).

---

Thunnini is an experimentation library to study and understand fundamental aspects of fine-tuners for neural sequence predictors. Currently, 9 different tuners, such as soft prompting, embedding tuning, or low-rank adaptation (LoRA), are implemented on two neural architectures (LSTMs and transformers). Thunnini is also the name of the zoological tribe of the tunas (17 species).

!Taxonomy of fine tunas for neural sequence predictors

Thunnini provides functionality to:

1. Pretrain neural sequential predictors via log loss minimization over samples from a data generator. 2. Fine-tune neural models to a target data distribution. Tune either weights (original weights, or additional weights, like in LoRA), or tune a prompt prefix (e.g., Soft Prompting). 3. Evaluate tuned models' prediction performance on one or more evaluation data distributions. 4. Compare different architectures and fine-tuning methods, and compare against baselines: the optimal oracle predictor, several exact Bayesian predictors, and the untuned neural predictors.

*Simple experimentation:* Standard Thunnini experiments are very simple to specify via a set of configurations (plain-text dicts or dataclasses) for data generation, model architecture, training, and fine-tuning procedures, as well as evaluation settings. Full reproducibility, given the same configuration, is ensured.

*Batteries included:* Thunnini comes with easily configurable LSTMs and transformers (decoder-only) and some data generators (coin-flip or dice-roll sources, for which the exact Bayesian predictor is tractable). Thunnini also comes with a notebook to run a full experimentation pipeline (pretraining, fine-tuning, evaluation and comparison) by setting a few lines of configuration.

*Easily extendable:* Thunnini can be extended with more predictors and data generators by implementing the respective interfaces and passing all tests.

The design philosophy is to provide a nimble experimentation library for studying conceptual aspects of fine-tuning methods and prompting untrained networks, enabled by having full understanding and control over the data distributions. Connections to the theory of meta-learning and Bayesian sequential prediction from SGD-based log loss minimization, can easily be verified empirically (Genewein et al. 2025). Thunnini aims at models and data that train or fine-tune on a single GPU within minutes. LLM-scale experiments are (far) beyond the scope of Thunnini.

Usage

The fastest way to get started with Thunnini is via Google's Colab, where you can directly run notebooks in the cloud (the colabs will automatically clone Thunnini's git repo when running on Colab). You can use the following links to open the Full Experiment notebook or the Demo notebook on Colab. Note that it is recommended that you use a GPU runtime, both locally and on Colab. If no GPU runtime is available small experiments can be run on a CPU by adjusting experiment settings (shorter sequences, smaller models, fewer training/tuning steps, less repetitions).

If you want to use thunnini locally, e.g., because you want to modify or extend the code or run python scripts instead of notebooks, use the instructions in the next section. See also [how to run Thunnini's tests](#running-tests).

Local installation and usage

Clone the source code into a local directory:

git clone https://github.com/google-deepmind/thunnini.git
cd thunnini

This repository requires Python 3.11. pip install -r requirements.txt will install all required dependencies. This is best done inside a virtual Python environment. To that end, install virtualenv:

sudo apt-get install virtualenv python3-venv

Then, create and activate the virtualenv:

virtualenv thunnini_env
source thunnini_env/bin/activate

Alternatively you can also use conda to manage virtual environments. See the conda documentation for instructions.

Inside your virtual environment, use pip to install all required dependencies:

pip install -r requests
pip install -r requirements.txt

Running notebooks locally: To get started with Thunnini locally, start a local Jupyter notebook server.

> If you followed the installation instructions above, make sure to first > activate your virtual environment and set the PYTHONPATH. Change dir to the > local clone of the thunnini folder and run: > > ``bash > source thunnini_env/bin/activate > export PYTHONPATH=$(pwd)/.. >

Start a local Jupyter notebook with:

jupyter notebook

This will give you a (local) URL in the terminal with an authentication token which you need to click on or copy and paste into your browser. From there, navigate to thunnini/colabs and open one of the two notebooks. Alternatively, the local notebook server can be set as a local runtime for Colab. See [local Colab…