google-deepmind/sonnet

google-deepmind/sonnet

Description: TensorFlow-based neural network library

Language: Python

License: Apache-2.0

Stars: 9922

Forks: 1309

Open issues: 43

Created: 2017-04-03T11:34:35Z

Pushed: 2026-05-06T20:18:31Z

Default branch: v2

Fork: no

Archived: no

README: !Sonnet

Sonnet

**Documentation** | [Examples](#examples)

Sonnet is a library built on top of TensorFlow 2 designed to provide simple, composable abstractions for machine learning research.

Introduction

Sonnet has been designed and built by researchers at DeepMind. It can be used to construct neural networks for many different purposes (un/supervised learning, reinforcement learning, ...). We find it is a successful abstraction for our organization, you might too!

More specifically, Sonnet provides a simple but powerful programming model centered around a single concept: snt.Module. Modules can hold references to parameters, other modules and methods that apply some function on the user input. Sonnet ships with many predefined modules (e.g. snt.Linear, snt.Conv2D, snt.BatchNorm) and some predefined networks of modules (e.g. snt.nets.MLP) but users are also encouraged to build their own modules.

Unlike many frameworks Sonnet is extremely unopinionated about how you will use your modules. Modules are designed to be self contained and entirely decoupled from one another. Sonnet does not ship with a training framework and users are encouraged to build their own or adopt those built by others.

Sonnet is also designed to be simple to understand, our code is (hopefully!) clear and focussed. Where we have picked defaults (e.g. defaults for initial parameter values) we try to point out why.

Getting Started

Examples

The easiest way to try Sonnet is to use Google Colab which offers a free Python notebook attached to a GPU or TPU.

• Predicting MNIST with an MLP
• Training a Little GAN on MNIST
• Distributed training with `snt.distribute`

Installation

To get started install TensorFlow 2.0 and Sonnet 2:

$ pip install tensorflow tensorflow-probability
$ pip install dm-sonnet

You can run the following to verify things installed correctly:

import tensorflow as tf
import sonnet as snt

print("TensorFlow version {}".format(tf.__version__))
print("Sonnet version {}".format(snt.__version__))

Using existing modules

Sonnet ships with a number of built in modules that you can trivially use. For example to define an MLP we can use the snt.Sequential module to call a sequence of modules, passing the output of a given module as the input for the next module. We can use snt.Linear and tf.nn.relu to actually define our computation:

mlp = snt.Sequential([
snt.Linear(1024),
tf.nn.relu,
snt.Linear(10),
])

To use our module we need to "call" it. The Sequential module (and most modules) define a __call__ method that means you can call them by name:

logits = mlp(tf.random.normal([batch_size, input_size]))

It is also very common to request all the parameters for your module. Most modules in Sonnet create their parameters the first time they are called with some input (since in most cases the shape of the parameters is a function of the input). Sonnet modules provide two properties for accessing parameters.

The variables property returns all tf.Variables that are referenced by the given module:

all_variables = mlp.variables

It is worth noting that tf.Variables are not just used for parameters of your model. For example they are used to hold state in metrics used in snt.BatchNorm. In most cases users retrieve the module variables to pass them to an optimizer to be updated. In this case non-trainable variables should typically not be in that list as they are updated via a different mechanism. TensorFlow has a built in mechanism to mark variables as "trainable" (parameters of your model) vs. non-trainable (other variables). Sonnet provides a mechanism to gather all trainable variables from your module which is probably what you want to pass to an optimizer:

model_parameters = mlp.trainable_variables

Building your own module

Sonnet strongly encourages users to subclass snt.Module to define their own modules. Let's start by creating a simple Linear layer called MyLinear:

class MyLinear(snt.Module):

def __init__(self, output_size, name=None):
super(MyLinear, self).__init__(name=name)
self.output_size = output_size

@snt.once
def _initialize(self, x):
initial_w = tf.random.normal([x.shape[1], self.output_size])
self.w = tf.Variable(initial_w, name="w")
self.b = tf.Variable(tf.zeros([self.output_size]), name="b")

def __call__(self, x):
self._initialize(x)
return tf.matmul(x, self.w) + self.b

Using this module is trivial:

mod = MyLinear(32)
mod(tf.ones([batch_size, input_size]))

By subclassing snt.Module you get many nice properties for free. For example a default implementation of __repr__ which shows constructor arguments (very useful for debugging and introspection):

>>> print(repr(mod))
MyLinear(output_size=10)

You also get the variables and trainable_variables properties:

>>> mod.variables
(,
)

You may notice the my_linear prefix on the variables above. This is because Sonnet modules also enter the modules name scope whenever methods are called. By entering the module name scope we provide a much more useful graph for tools like TensorBoard to consume (e.g. all operations that occur inside my_linear will be in a group called my_linear).

Additionally your module will now support TensorFlow checkpointing and saved model which are advanced features covered later.

Serialization

Sonnet supports multiple serialization formats. The simplest format we support is Python's pickle, and all built in modules are tested to make sure they can be saved/loaded via pickle in the same Python process. In general we discourage the use of pickle, it is not well supported by many parts of TensorFlow and in our…