Eval Driven Development With Claude Code

LLM Eval Driven Development with Claude Code

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Eval Driven Development With Claude Code LLM Eval Driven Development with Claude Code

PUBLISHED 8/25/2025

Table of Contents

Evolving Our Workflow: Supercharging Claude with MCP Servers

Prompting Our New Development Flow

Building and Expanding the Test Suite

Evolving the TDD Workflow

Table of Contents

In our previous blog , we showed how to go from one test to many tests with Eval Protocol with Cursor. But what if you're starting from scratch? Today, with Claude Code supercharged by MCP servers pointing directly to our docs and a deep wiki, we'll show you how to go from 0 to 1. In other words, from a completely blank project to your first fully tested AI agent. To recap the core idea from the previous blog, we're adapting the classic software engineering practice of Test-Driven Development (TDD) to use evals in the era of LLMs. The idea is simple: you write evals that define the desired behavior before writing the actual code, and then build your agent to pass them. This post will demonstrate how applying a TDD workflow ensures that as you add new features or swap out models, you have a safety net to prevent regressions. Evolving Our Workflow: Supercharging Claude with MCP Servers

To give our AI agent the context it needs about Eval Protocol by Fireworks AI, we're leveraging Model Context Protocol (MCP) servers. These act as secure bridges, allowing the agent to access external data sources. With Claude Code, you can easily add these servers directly from the command line: 1 2 claude mcp add --transport http eval-protocol-docs https://evalprotocol.io/mcp claude mcp add --transport http eval-protocol-deep-wiki https://mcp.deepwiki.com/mcp

Claude code MCP setup screenshot The first link is an MCP server that points to our documentation, and the second link is an MCP server that points to our GitHub repository of our open-source implementation of Eval Protocol. These MCP servers give Claude the ability to "read the manual.” Alternatively, you can use an mcp.json file. This is a commonly used format that other MCP clients, like Cursor, can also use, making your configurations portable across different tools. 1 2 3 4 5 6 7 8 9 10 { "mcpServers" : { "eval-protocol-docs" : { "url" : "https://evalprotocol.io/mcp" } , "eval-protocol-deep-wiki" : { "url" : "https://mcp.deepwiki.com/mcp" } } }

This simple addition is a game-changer, as it grounds the agent in the specific documentation relevant to our task. Prompting Our New Development Flow

With our environment enhanced, the next step is to give Claude Code its mission. We start with a general prompt that defines its role and how to use Eval Protocol. Below that, we append the specific instructions for our project, which are copied over from our previous blog's prompt . This "meta-prompting" is crucial for guiding the agent effectively. Here is the initial prompt we'll use with Claude Code: You are an applied AI engineer whose job is to write tests called "evals" in the form of code. An "eval" helps determines whether an AI application is working as expected by programmatically assessing the output of the model. To do this, you will use a library called "eval-protocol" (aka EP) that helps you easily author, run, and review evals. Evals accept whats called an EvaluationRow and outputs an EvaluationRow (or multiple EvaluationRows depending on the mode of the @evaluation_test decorator). In the eval, a score from 0 to 1 is generated based on the output of the model. Use the provided tools to help you understand more about eval-protocol so that you can generate evals for the given task. The tools can help provide examples of evals, guide you with tutorials on how to use eval-protocol, retrieve reference documentation for the eval-protocol API, and ask questions about the source code of eval-protocol. GitHub source repos:

• for the docs, see eval-protocol/eval-protocol
• for the Python SDK, see eval-protocol/python-sdk

Please follow the below instructions to write our evals: project.md is something I want to work on, I want you to use EP to create the application. We want to use https://github.com/lerocha/chinook-database has the database info, please use https://github.com/gldc/mcp-postgres for the setup, then let's setup the project with the MCP server, and then use Eval Protocol to test at least 1 simulated user to check if the project is working end to end, thanks! Claude code screenshot To be clear, you would add your own instructions as the last paragraph. This prompt does several key things: Assigns a Persona: It tells Claude to act as an "applied AI engineer." Defines the Core Task: It explains what an "eval" is and its purpose. Introduces the Tools: It points to the Eval Protocol library and the MCP servers we configured. Provides Specific Resources: It gives direct links to the GitHub repositories for context. States the Goal: It clearly outlines the end-to-end task of setting up the project and creating the first test.

Note: Enabling Web Access: Our prompt asks Claude to reference several GitHub URLs, so it needs web access to succeed. You can configure this in Claude Code's settings (e.g., in .claude/settings.json ), by adding a rule to permit the WebFetch tool.

Building and Expanding the Test Suite

From there, because we supercharged it with the MCP tools, our AI agent Claude Code was able to navigate and write Eval Protocol tests more accurately. It set up an environment and had our first test running quickly. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 @evaluation_test ( input_dataset = [ "data/storefront_eval_dataset.jsonl" ] , dataset_adapter = storefront_dataset_to_evaluation_row , completion_params = [ { "model" : "fireworks_ai/accounts/fireworks/models/gpt-oss-120b" , "temperature" : 0.8 } ] , passed_threshold = 0.6 , rollout_processor = AgentRolloutProcessor ( ) , mcp_config_path = "mcp_server_config.json" , num_runs = 1 , mode = "pointwise" ) def test_chinook_storefront_evaluation ( row : EvaluationRow ) - > EvaluationRow :

For a bit more color, this is what the initial data/storefront_eval_dataset.jsonl looked like. 1 2 3 4 { "id" : "browse_only_search" , "prompt" : "Find bossa nova songs under $0.99." , "expected_behaviors" : [ "browse_only_mode" , "applies_limit" , "price_filter" , "genre_search" ] ,...