NVIDIA/Judges-Verdict

NVIDIA/Judges-Verdict

Description: Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement

Language: Python

License: Apache-2.0

Stars: 15

Forks: 4

Open issues: 0

Created: 2025-10-06T13:05:23Z

Pushed: 2025-10-08T17:23:31Z

Default branch: main

Fork: no

Archived: no

README:

Judge's Verdict

A comprehensive framework for evaluating Large Language Models (LLMs) as judges for assessing the quality of AI-generated responses.

Table of Contents

• [Overview](#overview)
• [Features](#features)
• [Installation](#installation)
• [Project Structure](#project-structure)
• [Quick Start](#quick-start)
• [Configuration](#configuration)
• [Command Line Options](#command-line-options)
• [Troubleshooting](#troubleshooting)
• [Extending the Framework](#extending-the-framework)
• [Development](#development)
• [Contributing](#contributing)

Overview

This repository contains the evaluation pipeline for running Judge's Verdict on the nvidia/judges-verdict HuggingFace dataset. The framework uses LiteLLM as the officially supported approach for interfacing with various LLM providers through a unified API.

Supported Providers via LiteLLM:

• OpenAI (GPT-4, GPT-4o, etc.)
• Anthropic (Claude models)
• NVIDIA NIM (Llama, Nemotron, Gemma, etc.)
• Local models (via vLLM or other compatible servers)
• Many other providers (see LiteLLM docs)

Features

• Flexible Judge Configuration: YAML-based configuration system for managing multiple judge models
• Multiple Serving Frameworks: Support for various LLM serving backends
• Annotation Preparation: Scripts for preparing and merging human annotations
• Parallel Processing: Efficient evaluation with configurable workers
• Extensible Architecture: Easy to add new judge models and metrics

Installation

# Clone the repository
git clone https://github.com/NVIDIA/Judges-Verdict.git
cd Judges-Verdict

# Install using Poetry (recommended)
poetry install

# Or install with pip
pip install -e .

Project Structure

Judges-Verdict/
├── llm_judge_benchmark/ # Core package
│ ├── scoring/ # Judge scoring implementation
│ ├── utils/ # Utility functions
│ ├── judge_config.py # Judge configuration classes
│ └── judge_config_manager.py # YAML config management
├── scripts/ # Standalone scripts
│ └── data_prep.py # Data preparation script (fetches from HuggingFace)
├── config/ # Configuration files
│ ├── judge_config_litellm.yaml
│ └── judge_config_litellm_example.yaml
├── data/ # Data directory
│ ├── annotations_sample.json # Sample annotations
│ └── evaluation.json # Merged evaluation dataset (created by data_prep.py)
├── results/ # Evaluation results
└── docs/ # Documentation

Quick Start

1. Install dependencies

# Using Poetry (recommended)
poetry install

# Or using pip
pip install -e .

2. Set up API keys

The project uses the LiteLLM framework as the officially supported approach for judge scoring, which provides a unified interface for multiple LLM providers.

# For OpenAI models (gpt-4o, gpt-4, etc.)
export OPENAI_API_KEY="your-openai-api-key"

# For Anthropic models (claude-sonnet-4, etc.)
export ANTHROPIC_API_KEY="your-anthropic-api-key"

# For NVIDIA NIM models (llama, nemotron, gemma, etc.)
export NVIDIA_NIM_API_KEY="your-nvidia-nim-api-key"
export NVIDIA_NIM_API_BASE="https://integrate.api.nvidia.com/v1" # or your custom endpoint

# For other providers supported by LiteLLM, see: https://docs.litellm.ai/docs/providers

Note: The LiteLLM framework automatically handles provider-specific authentication and API formatting.

3. Prepare the data

# Run the data preparation script
python scripts/data_prep.py

This will:

• Download the nvidia/judges-verdict dataset from HuggingFace
• Process the train split (200 techqa entries, annotations removed)
• Process the test split (1,794 entries across 6 datasets):
• coral (318 entries): Joins with CORAL dataset from HuggingFace
• dc767 (347 entries): Joins with DC767 CSV from GitHub
• enterprise_rag_benchmark (346 entries): Used as-is
• hotpotqa (342 entries): Used as-is
• squad (346 entries): Used as-is
• techqa (95 entries): Used as-is
• Merge all datasets into data/evaluation.json (1,994 total entries)

4. Run Judge Scoring

# Using the installed script
poetry run llm-judge-score \
--evaluation-file data/evaluation.json \
--output-dir results/

# Or run directly
python -m llm_judge_benchmark.scoring.llm_judge_scoring \
--evaluation-file data/evaluation.json \
--config config/judge_config_litellm.yaml \
--output-dir results/

Configuration

Judge Configuration

Judges are configured using YAML files with LiteLLM as the primary framework. Each judge requires:

• identifier: Unique identifier for the judge (it's also used as the folder name for judge evaluation results)
• framework: Always use litellm for the officially supported approach
• model: Model name in LiteLLM format (e.g., openai/gpt-4o, anthropic/claude-sonnet-4-20250514, nvidia_nim/meta/llama-3.1-70b-instruct)
• temperature: Temperature setting (usually 0.0 for consistency)
• max_tokens: Maximum tokens for response
• num_workers: Number of parallel workers
• timeout: Request timeout in seconds

The model identifiers in the configuration YAML will be the folder names where results will be stored. For example:

models:
gpt-4o: # This creates results in results/gpt-4o/
framework: litellm
model: openai/gpt-4o

meta_llama-3.1-70b-instruct: # This creates results in results/meta_llama-3.1-70b-instruct/
framework: litellm
model: nvidia_nim/meta/llama-3.1-70b-instruct

Local Model Configuration

Local models are also supported by the litellm framework. When configuring a local model, you need to specify:

• framework: litellm
• model: The model name in litellm format (e.g., hosted_vllm/...)
• base_url: The URL of your local model server
• api_key: Usually set to "EMPTY" for local models

Example configuration for a local Qwen model:

local/qwen-0.5b:
framework: litellm
model: hosted_vllm/Qwen/Qwen2-0.5B-Instruct
base_url: http://localhost:8000/v1
api_key: EMPTY
num_workers: 1

This assumes you have a VLLM server running locally on port 8000 serving the Qwen2-0.5B-Instruct model.

The repository includes configuration files for the LiteLLM framework: -...