amazon-science/Automating-Relational-Machine-Learning

amazon-science/Automating-Relational-Machine-Learning

Language: Python

License: NOASSERTION

Stars: 5

Forks: 0

Open issues: 19

Created: 2026-02-26T01:02:30Z

Pushed: 2026-05-09T04:19:37Z

Default branch: main

Fork: no

Archived: no

README:

(ICLR 2026) Relatron: Automating Relational Machine Learning over Relational Databases

Zhikai Chen, Han Xie, Jian Zhang, Jiliang Tang, Xiang Song, Huzefa Rangwala

Abstract

Predictive modeling over relational databases (RDBs) powers applications in various domains, yet remains challenging due to the need to capture both cross-table dependencies and complex feature interactions. Recent Relational Deep Learning (RDL) methods automate feature engineering via message passing, while classical approaches like Deep Feature Synthesis (DFS) rely on predefined non-parametric aggregators. Despite promising performance gains, the comparative advantages of RDL over DFS and the design principles for selecting effective architectures remain poorly understood.

We present a comprehensive study that unifies RDL and DFS in a shared design space and conducts large-scale architecture-centric searches across diverse RDB tasks. Our analysis yields three key findings: (1) RDL does not consistently outperform DFS, with performance being highly task-dependent; (2) no single architecture dominates across tasks, underscoring the need for task-aware model selection; and (3) validation accuracy is an unreliable guide for architecture choice. This search yields a curated model performance bank that links model architecture configurations to their performance; leveraging this bank, we analyze the drivers of the RDL-DFS performance gap and introduce two task signals -- RDB task homophily and an affinity embedding that captures size, path, feature, and temporal structure -- whose correlation with the gap enables principled routing. Guided by these signals, we propose Relatron, a task embedding-based meta-selector that first chooses between RDL and DFS and then prunes the within-family search to deliver strong performance. Lightweight loss-landscape metrics further guard against brittle checkpoints by preferring flatter optima. In experiments, Relatron resolves the "more tuning, worse performance" effect and, in joint hyperparameter-architecture optimization, achieves up to 18.5% improvement over strong baselines with 10x lower computational cost than Fisher information-based alternatives.

Environment Setup

This project uses pixi for environment and dependency management. All Python and system dependencies (including CUDA-enabled PyTorch, PyG, and RAPIDS) are specified in pixi.toml.

Install pixi

curl -fsSL https://pixi.sh/install.sh | bash

Install dependencies

pixi install -e iclr

This creates an isolated environment with Python 3.12, PyTorch 2.6.0 (CUDA 12.4), PyTorch Geometric, RAPIDS, and all required packages.

Activate the environment

pixi shell -e iclr

All commands below should be run inside the pixi shell (or prefixed with pixi run -e iclr).

Repository Structure

Relatron/
├── configs/ # Hydra/YAML configuration files
│ └── default/ # Default task and model configs
├── data/ # Data loading and dataset classes
│ ├── relbench.py # RelBench data loader
│ ├── mtaskdataset.py # Multi-task tabular dataset
│ └── gboltdataset.py # Graph-based online learning dataset
├── dbinfer/ # Database inference and preprocessing
│ ├── preprocess/ # Data transforms (e.g., float compression)
│ └── solutions/ # Solution backends (TabPFN, etc.)
├── models/ # Model implementations
│ ├── nn/ # Neural network architectures (AutoGNN, encoder)
│ ├── dfs/ # Deep Feature Synthesis pipeline
│ ├── heads/ # Prediction heads
│ ├── automl/ # AutoTransfer and loss-landscape utilities
│ ├── embedder.py # Feature embedder
│ └── regressor.py # Regression model wrapper
├── swap/ # Architecture search and HPO engine
│ ├── hpo.py # Task-embedding HPO (Relatron core)
│ ├── execution.py # Training execution wrapper
│ ├── generate.py # Search space generation
│ ├── heuristics.py # Task heuristic signal computation
│ ├── heuristics_at.py # AutoTransfer heuristics
│ ├── regression.py # Meta-regression models
│ └── search_and_plot.py# Search result analysis and plotting
├── utils/ # Shared utilities
│ ├── information.py # Task lists and dataset metadata
│ ├── heuristics.py # Homophily and task signal computation
│ ├── hpo.py # HPO catalog and objective helpers
│ ├── graph.py # Graph construction utilities
│ └── database.py # Database helpers
├── scripts/ # Experiment and data scripts
│ ├── download_and_generate_dfs.py # Download datasets + generate DFS features
│ ├── exp31.py # Figure 3-1: rank comparison plot
│ ├── exp42.py # Table 4 & 5: task signal evaluation
│ └── run_hpo.py # HPO experiments (Tables 4 & 5)
├── plot/ # Plotting and analysis scripts
├── results/ # Output CSVs and figures
├── relbench/ # Local fork of RelBench (editable install)
├── pytorch-frame/ # Local fork of PyTorch Frame (editable install)
├── rustler/ # Rust acceleration module (editable install)
└── pixi.toml # Environment and dependency specification

Downloading Datasets and Generating DFS Features

Use the provided script to download all RelBench datasets and generate DFS features:

# Download all datasets and generate DFS features
pixi run -e iclr python scripts/download_and_generate_dfs.py

# Download a single dataset
pixi run -e iclr python scripts/download_and_generate_dfs.py --dataset rel-f1

# Download a single dataset-task pair
pixi run -e iclr python scripts/download_and_generate_dfs.py --dataset rel-f1 --task driver-dnf

# Download only (skip DFS generation)
pixi run -e iclr python scripts/download_and_generate_dfs.py --download-only

# Generate DFS only (assume datasets are already cached)
pixi run -e iclr python scripts/download_and_generate_dfs.py --dfs-only

Downloaded data is cached under cache_data/, and DFS features are saved to cache_data/old_dfs/.

Reproducing Experimental Results

Figure 3-1: Rank Comparison (Classification vs. Regression)

This generates the rank comparison bar charts from the model performance bank results in results/result31.csv:

pixi run -e iclr python scripts/exp31.py

Output is saved to plot/result31.png and plot/result31.pdf.

Table 4 & Table 5: Task Signal Evaluation and Meta-Selector

Run the task signal feature…