google-deepmind/hope

google-deepmind/hope

Description: Hilbert Operator for Progressive Encoding (HOPE), and Dispersed Elastic Fine-Tuning (DEFT).

Language: Python

License: Apache-2.0

Stars: 3

Forks: 0

Open issues: 0

Created: 2026-10-27T19:55:42Z

Pushed: 2026-10-27T21:25:09Z

Default branch: main

Fork: no

Archived: no

README:

HOPE: Hilbert Operator for Progressive Encoding & DEFT Transfer Learning

This repository contains the reference implementations of HOPE (*Hilbert Operator for Progressive Encoding*) and DEFT (*Dispersed Elastic Fine-Tuning*):

1. Data-Free Progressive Model Compression (`hope.py`, `compress_resnet50.py`, `hope_baselines.py`, `plot_compression_curves.py`): HOPE compresses an ImageNet-pretrained ResNet-50 (Keras) through a sequence of analytical steps (pruning a neuron via PRUNE, merging two neurons into one synthesized parent neuron via MERGE, or evicting the residual branch of a bottleneck block via EVICT), selected greedily by a rate-distortion criterion. Every step is recorded together with the information needed to undo it. The resulting *curriculum* lets the compressed network be decoded step by step to any intermediate density, or all the way back to the pretrained network (lossless up to float32 round-off). HOPE is strictly data-free: every cost is evaluated in closed form from the weights and the stored BatchNorm parameters and statistics. It uses no training images, no calibration data, and no fine-tuning; ImageNet is only needed to measure the accuracy of the compressed networks. On ResNet-50, HOPE operates on the 16 bottleneck blocks (32 quadruplets, 7,552 neurons); the full curriculum has 3,066 steps (2,983 prunes, 67 merges, 16 block evictions) and goes from density 1.0 down to 0.1897, where every residual branch has been evicted. 2. Cross-Domain Transfer Learning via Dispersed Elastic Fine-Tuning (`deft.py`): DEFT uses HOPE's data-free progressive encoding curriculum on a pretrained source network to partition each hidden layer into a frozen Universal Core ($\mathcal{F}_{\mathrm{core}}$) and a plastic Slack ($\mathcal{F}_{\mathrm{slack}}$). By severing upstream-slack-to-downstream-core connections at initialization ($t = 0$) and restricting gradient updates to slack-destination parameters during target fine-tuning ($t > 0$), DEFT bounds the one-time initialization shock by the capacity of the slack subspace and guarantees exact zero dynamic forgetting on the source task across all fine-tuning epochs (see [Cross-domain transfer learning with DEFT (deft.py)](#cross-domain-transfer-learning-with-deft-deftpy)).

For full derivations, see *Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks* ([Citation](#citation)).

Method

HOPE treats each hidden neuron of a bottleneck block (a convolution filter with its BatchNorm (BN), its ReLU, and its slice of the next convolution) as a rank-1 Hilbert–Schmidt operator $f_i = g_i \otimes w_{\mathrm{out},i}$. The norm of this operator (the neuron's *capacity*) and the inner products between neurons are computed in closed form under a Gaussian surrogate built from the stored BN parameters, under which the pre-activation of neuron $i$ is $y_i \sim \mathcal{N}(\beta_i, \gamma_i^2)$. Three actions are available:

| Action | What it does | Cost $\mathcal{J}$ | |---|---|---| | PRUNE | removes one neuron $i$ | $\mathcal{J}^\circ_{\mathrm{prune}}(i) = \dfrac{N\,\lVert f_i \rVert_\circ}{E_a - \lVert f_i \rVert_\circ}$ | | MERGE | replaces two neurons $(i, j)$ of a layer by one synthesized parent neuron $f_p = s^{\ast}\psi^{\ast}$ | $\mathcal{J}^\circ_{\mathrm{merge}}(i,j) = \dfrac{N\sqrt{\lVert f_i - f_p \rVert_\circ^2 + \lVert f_j - f_p \rVert_\circ^2}}{E_a - \lVert f_i \rVert_\circ - \lVert f_j \rVert_\circ + \lVert f_p \rVert_\circ}$ | | EVICT | removes the residual branch of a bottleneck block (internal layers $l \in \{1, 2\}$) | $\mathcal{J}_{\mathrm{evict}} = \sum_{l \in \{1,2\}} N_l\,\dfrac{E_a^{(l)}}{E_{\mathrm{init}}^{(l)}}$ |

At every step the encoder executes the admissible action with the smallest distortion rate $\mathrm{DR} = \mathcal{J} / \Delta P^{\mathrm{init}}$, the cost per released parameter, until no action is left. Each executed action is stored as an InstructionPacket; the packets in reverse order form the curriculum.

Components of the method:

• Closed-form ReLU kernels. With $c_i = \beta_i / |\gamma_i|$, the mean

$m_i = |\gamma_i|\phi(c_i) + \beta_i\Phi(c_i)$ and second moment $K(i,i) = (\gamma_i^2 + \beta_i^2)\Phi(c_i) + \beta_i|\gamma_i|\phi(c_i)$ of each ReLU activation have closed forms, and the covariance $C(i,j) = K(i,j) - m_i m_j$ of two biased ReLUs is evaluated exactly through the bivariate normal CDF $\Phi_2(x, y; \rho)$, computed with Owen's T function.

• Mean-centred costs and parent synthesis. Capacities

$\lVert f_i \rVert_\circ = \lVert w_{\mathrm{out},i} \rVert\sqrt{v_i}$, prune and merge costs, and the synthesis of merge parents are computed on the zero-mean fluctuation subspace $\mathcal{H}_0^\perp$, i.e. with the centred kernels $v_i = C(i,i) = K(i,i) - m_i^2$ and $C(i,j) = K(i,j) - m_i m_j$.

• Optimal parent neuron. A merge writes a parent neuron

$f_p = s^{\ast}\psi^{\ast}$ whose input direction $u_c$, output direction $v^{\ast}$, optimal scale $s^{\ast}$ and physical parameters (raw filter, BN $\gamma_p, \beta_p, \mu_p, \sigma_p^2$) are computed in closed form.

• Consumer BatchNorm mean compensation. The mean shift that a prune or merge

causes at the input of the downstream BN is absorbed into that BN's running mean ($\mu^b \leftarrow \mu^b - m_i \sum_t \chi_t w_{t,i}$ for a prune; $\mu^b\leftarrow\mu^b+\sum_t\chi_t(m_p w'_{t,p}-m_i w_{t,i}-m_j w_{t,j})$ for a merge), with zero-padding tap fractions $\chi_t$ for 3×3 consumers.

• Coherence-weighted block eviction. When a residual branch is evicted, its

kernels and BN tensors are zeroed except the $\beta$ of its last BN, which keeps the coherence-weighted Wiener shift $\delta_3^{\ast} = \alpha_3 \beta_3$ with $\alpha_3 = \beta_3^2 / (\gamma_3^2 + \beta_3^2)$.

• Rate-distortion greedy selection. All candidate actions of all layers

compete through their distortion rate $\mathrm{DR} = \mathcal{J} / \Delta P^{\mathrm{init}}$; merge candidates are scored in $O(1)$ per pair from a decoupled all-pairs cache, and each executed action updates only the...