amazon-science/ListQA-Eval

amazon-science/ListQA-Eval

Language: Python

License: NOASSERTION

Stars: 1

Forks: 0

Open issues: 0

Created: 2026-10-12T22:56:22Z

Pushed: 2026-10-13T02:49:03Z

Default branch: main

Fork: no

Archived: no

README:

ListQA-Eval

Official evaluation code for [ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models](https://listqa.github.io) (NeurIPS 2026, Evaluations and Datasets Track).

Project page · Dataset on Hugging Face

>[!NOTE] >This code is being released solely for academic and scientific reproducibility purposes, in support of the methods and findings described in the associated publication. Pull requests are not being accepted in order to maintain the code exactly as it was used in the paper.

ListQA measures how well language models produce structured, list-formatted answers to factoid questions. It evaluates both the content accuracy (are the list elements factually correct?) and format compliance (does the output follow the requested single-faceted or multi-faceted list structure?) using multiple complementary metrics — from lightweight string matching to full LLM-as-a-Judge grading.

---

Table of Contents

• [Overview](#overview)
• [Features](#features)
• [Installation](#installation)
• [The ListQA Dataset](#the-listqa-dataset)
• [Dataset Format](#dataset-format)
• [Generating Predictions](#generating-predictions)
• [Quick Start](#quick-start)
• [Evaluation Metrics](#evaluation-metrics)
• [Relaxed Exact Match](#1-relaxed-exact-match-precision--recall)
• [ROUGE List Match](#2-rouge-list-element-match)
• [List Format Compliance](#3-list-format-compliance)
• [LLM-as-a-Judge](#4-llm-as-a-judge)
• [LLM-as-a-Judge Details](#llm-as-a-judge-details)
• [Python API](#python-api)
• [CLI Reference](#cli-reference)
• [Output Format](#output-format)
• [Running Tests](#running-tests)
• [Project Structure](#project-structure)
• [License](#license)
• [Citation](#citation)

---

Overview

ListQA evaluates list-formatted question answering, where models must generate structured numbered lists in response to factoid questions. Questions come in two flavours:

• Single-faceted — simple lists where each element is a standalone fact:

1. Alabama A&M
2. Grambling State
3. Savannah State
4. Southeast Missouri State

• Multi-faceted — complex lists where each element contains sub-categories:

1. 2018
* Name of the work: "IDGAF"

2. 2020
* Name of the work: "Physical"

The evaluation pipeline: 1. Parses predicted and reference lists into individual elements 2. Matches predicted elements to reference elements using ROUGE + Hungarian assignment 3. Grades each matched pair using an LLM-as-a-Judge (locally via HuggingFace Transformers) 4. Aggregates scores into precision, recall, and F1 with optional metadata breakdowns

---

Features

• Zero proprietary dependencies — completely standalone, no ties to any internal evaluation framework
• Local LLM-as-a-Judge via HuggingFace transformers.AutoModelForCausalLM — runs on CPU or GPU, no API keys or cloud services required
• Multiple metric types:
• Relaxed exact-match precision & recall (substring containment)
• ROUGE-based list element matching with optimal Hungarian assignment
• List format compliance checking (single-faceted vs multi-faceted)
• LLM-as-a-Judge element-level grading with aggregated P/R/F1
• Flexible data loading — local JSONL files or HuggingFace Hub datasets
• Metadata-sliced breakdowns by country, category, and year range
• CLI and Python API for easy integration into research workflows
• Comprehensive test suite — 56 tests with real dataset samples and mocked judge

---

Installation

# From the repository
git clone https://github.com/amazon-science/ListQA-Eval.git
cd ListQA-Eval

# Option 1: Install as editable package (recommended)
pip install -e ".[dev]"

# Option 2: Install from requirements.txt
pip install -r requirements.txt
pip install -e .

Requirements

| Dependency | Version | Purpose | |---|---|---| | Python | ≥ 3.9 | Runtime | | PyTorch | ≥ 2.0 | LLM judge inference | | HuggingFace Transformers | ≥ 4.35 | Model loading & generation | | HuggingFace Accelerate | ≥ 0.24 | Device management | | evaluate | ≥ 0.4 | ROUGE metric computation | | rouge-score | ≥ 0.1.2 | ROUGE scoring backend | | scipy | ≥ 1.10 | Hungarian assignment algorithm | | Jinja2 | ≥ 3.1 | Prompt template rendering | | datasets | ≥ 2.14 | HuggingFace Hub data loading | | tqdm | ≥ 4.65 | Progress bars |

---

The ListQA Dataset

The benchmark is hosted on Hugging Face at `AmazonScience/ListQA`. It has 9,045 human-curated questions grounded in Wikipedia tables, and every answer is a list of 3–10 elements.

| Config | Split(s) | Rows | Description | |---|---|---|---| | all (default) | test | 9,045 | Full benchmark | | simple | test | 5,378 | Single-faceted questions (SimpleListQA) | | complex | test | 3,667 | Multi-faceted questions (ComplexListQA) | | few_shot | simple / complex / combined | 14 / 10 / 12 | In-context example pools for few-shot prompting |

from datasets import load_dataset

ds = load_dataset("AmazonScience/ListQA", "simple", split="test")

The dataset has the questions and reference answers but no model outputs. You generate predictions first (see [Generating Predictions](#generating-predictions)) and then score them with this package.

---

Dataset Format

Input Data (JSONL)

Each line in the JSONL file must be a JSON object with these fields:

| Field | Required | Description | |---|---|---| | problem | ✅ | The question | | answer | ✅ | Gold reference answer (list-formatted) | | prediction | ✅ (for eval) | Model's generated answer | | metadata | ❌ | JSON string with optional metadata |

Metadata Schema

The optional metadata field is a JSON string that can contain:

{
"prompt_type": "single-faceted", // or "multi-faceted"
"category": "Sports", // topic category
"country": "USA", // country of origin
"year_range": "2018-2019", // temporal range
"ordered": false, // whether list order matters
"wikitable_source": "https://en.wikipedia.org/wiki/..." // source Wikipedia page
}

prompt_type is required for the format-compliance metric. The other fields are only used to...