microsoft/experiment-catalog

microsoft/experiment-catalog

Description: A comprehensive tool for cataloging, comparing, and analyzing experiment results. Experiment Catalog enables teams to track evaluation runs across projects, compare metrics against baselines, and identify performance regressions or improvements in AI and ML experimentation workflows.

Language: C#

License: MIT

Stars: 7

Forks: 3

Open issues: 10

Created: 2026-04-13T14:58:11Z

Pushed: 2026-09-21T10:58:48Z

Default branch: main

Fork: no

Archived: no

README: --- title: Experiment Catalog description: Catalog, compare, and analyze experiment runs with a .NET API, Svelte UI, and deterministic development harness. ms.date: 2026-09-02 ms.topic: overview ---

Experiment Catalog

A tool for cataloging, comparing, and analyzing experiment results. The Experiment Catalog enables teams to track evaluation runs across projects, compare metrics against baselines, and identify performance regressions or improvements in AI/ML experimentation workflows.

 

Overview

The Experiment Catalog is designed for teams running iterative experiments. It is particularly useful for AI evaluation pipelines where you need to:

• Track results across multiple evaluation runs
• Compare experiment metrics against established baselines
• Analyze performance trends and identify regressions
• Filter and drill down into specific ground-truth results
• Annotate experiments with links to commits, configurations, or documentation

Watch these walkthroughs:

• Installation (6:08)
• Usage (30:56)
• Configuration (16:36)

Architecture

The application consists of several main components:

| Component | Description | | ----------------- | --------------------------------------------------------------------------------- | | catalog | ASP.NET Core API that stores experiment data in Azure Blob Storage | | MCP interface | MCP tools hosted by the catalog API for project, experiment, and analysis actions | | cli | Python package and CLI for publishing metrics from CSV files or notebooks | | ui | Svelte frontend for visualizing and comparing experiments | | catalog.tests | xUnit tests for the catalog API |

Key Concepts

• A project is a collection of experiments sharing the same baseline, grounding data, and evaluation configuration. It typically aligns to a sprint. See [the experimentation process](./experimentation-process.md) for details.
• An experiment is a hypothesis-driven collection of evaluation runs within a project.
• A set is a group of results from a single evaluation run, also commonly called a permutation (for example, 3 iterations × 12 ground truths).
• A ref identifies a specific ground-truth entity being evaluated, allowing aggregation across iterations.
• A baseline is a reference point for comparison at the project or experiment level.

Features

Experiment Management

• Create projects and experiments with hypotheses
• Create projects, experiments, and push metric rows from the

[Experiment Catalog CLI](./cli/README.md) or the same Python API in notebooks

• Set project-level and experiment-level baselines
• Record arbitrary metrics without requiring metric definitions, while

optionally adding metric-definition descriptions for inline comparison context

• Annotate sets with commit hashes, configuration links, or notes

Comparison & Analysis

• Compare experiment results against baselines per set or per ref
• View aggregate statistics across sets
• Aggregate structured retrieval metrics with pooled or per-ref precision,

recall, and F1

• Surface response-only aggregate comparison metadata such as distinct

contributing refs and per-ref WIN/TIE counts against the experiment baseline

• Compute trusted runtime-only custom aggregate metrics from

administrator-managed Python files during comparison and meaningful-tags analysis (see the [Catalog README](./catalog/README.md#runtime-custom-aggregate-metrics))

• Drill down into individual ground-truth results
• Compare metrics across multiple evaluation runs
• Export raw metrics and direct-download artifact manifests at experiment or

set scope (see the [catalog API guide](./catalog/README.md#export-raw-metrics))

Filtering Capabilities

• Use the metrics filter to show or hide specific metrics in comparison views
• Use the tags filter to select ground truths by tags extracted from source data
• Use the free filter to write custom expressions that find specific results

Free Filter Examples

# Find poor performers
[generation_correctness] 20% better)
[generation_correctness] > [baseline.generation_correctness] * 1.2

# Find absolute metric differences
[generation_correctness] - [baseline.generation_correctness] > 0.05

# Find noisy aggregate metrics by standard deviation
result.metrics["generation_correctness"].std_dev > 0.10

# Find unstable aggregate metrics by coefficient of variation
result.metrics["generation_correctness"].coefficient_of_variation > 0.20

# Complex analysis - retrieval got worse but generation improved
[retrieval_recall] [baseline.generation_correctness]

# Find specific ground truths
ref == "TQ10" OR ref == "TQ25"

You can find out more about the Free Filter syntax and use cases in the [UI README](./ui/README.md#free-filter).

Getting Started

Prerequisites

• .NET 10 SDK
• Node.js 20+
• Python 3.10+ for the CLI, notebook

API, and optional runtime custom aggregate metrics

• Docker (for containerized deployment)
• Azure Storage Account
• Azure CLI (when using INCLUDE_CREDENTIAL_TYPES=azcli)

Running Locally

Experiment Catalog CLI

Install the reusable CLI and notebook package:

python3 -m venv cli/.venv
cli/.venv/bin/python -m pip install -e ./cli
source cli/.venv/bin/activate
export...