Test Driven Agent Development

Fireworks AI

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Test Driven Agent Development Test-Driven Agent Development with Eval Protocol

PUBLISHED 8/14/2025

Table of Contents 'Vibe' to a Concrete Plan Setup in minutes: vibes, Cursor, Eval Protocol

The First Test – From Zero to a Live Rollout From Vague Expectations to Concrete Tests

First Acceptance Test: "Find Jazz tracks under $0.99"

The Iterative Flow Step 5: Shifting Gears to Safety and Red Teaming The Power of This TDD Workflow

Table of Contents

Building AI agents is exciting, but let's be honest: they can be unpredictable. How do you add new features without secretly breaking old ones? How do you debug a complex, multi-turn conversation and prevent regressions? At Fireworks, we believe the answer lies in a familiar engineering practice:  Test-Driven Development (TDD) . We've developed the  Eval Protocol , a pytest-centric framework designed to bring structure and reliability to agent development. In this blog, we'll walk you through an end-to-end journey of building a digital store concierge agent. Using Cursor as our AI coding partner and Eval Protocol for testing, we'll show you how to go from a rough idea to a well-tested, trustworthy agent. 'Vibe' to a Concrete Plan

Every great project starts with an idea. I wanted to build a digital store concierge that could answer questions about the classic  Chinook’s music database . Instead of writing a detailed spec from scratch, I started with a high-level "vibe." I described my goal to an AI coding assistant (Cursor) and let it help me structure the project. This collaborative process produced a clear plan, which I saved as  project.md . This document became the blueprint for our test-driven journey. See the initial project plan here. Setup in minutes: vibes, Cursor, Eval Protocol

With a plan in hand, it was time to set up the development environment. My goal was to have: • A Postgres database (running the Chinook schema) accessible via a secure MCP (Model-Controller-Proxy) server in Docker. • A system prompt file to define the agent's persona and rules. • Eval Protocol to run, score, and visualize our tests.

I gave my AI assistant, Cursor, a single, comprehensive prompt to wire everything together: @project.md is something I want to work on, I want to to take a test drive development flow to create the application. @ https://evalprotocol.io/llms-full.txt has some details, there is also an MCP server for eval protocol. We want to use @ https://github.com/lerocha/chinook-database has the database info, please use @ https://github.com/gldc/mcp-postgres for the setup, then let's setup the project with the MCP server, and then test at least 1 simulated user to check if the project is working end to end, thanks! Cursor immediately generated the  mcp.json  config. This file tells Eval Protocol how to spin up the Postgres MCP server in a sandboxed Docker network, allowing the agent to safely interact with the database. 1 2 3 4 5 6 7 8 9 10 11 12 13 { "mcpServers" : { "postgres" : { "command" : "docker" , "args" : [ "run" , "-i" , "--rm" , "--network" , "digital_store_app_default" , "-e" , "DATABASE_URI=postgresql://postgres:postgres@chinook_db:5432/chinook" , "crystaldba/postgres-mcp" , "--access-mode=restricted" , "--transport=stdio" ] } } }

It also scaffolded our first pytest test using the Eval Protocol decorator. This powerful decorator links our test function to a dataset, an LLM, and our MCP configuration. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 @evaluation_test ( input_dataset = [ "tests/pytest/data/storefront_agent_browse.jsonl" ] , dataset_adapter = storefront_dataset_to_evaluation_row , model = [ "fireworks_ai/accounts/fireworks/models/gpt-oss-120b#pyroworks/wzflb9s1" ] , rollout_input_params = [ { "temperature" : 0.0 } ] , rollout_processor = default_agent_rollout_processor , mcp_config_path = "mcp.json" , passed_threshold = 0.9 , num_runs = 1 , max_concurrent_rollouts = 1 , mode = "pointwise" , ) def test_storefront_agent_browse ( row : EvaluationRow ) - > EvaluationRow : return EvaluationRow

The First Test – From Zero to a Live Rollout

We didn't need a perfect test to start. We created a tiny placeholder dataset and a test with no assertions. The goal was simple: see it run. 1 EP_PRINT_SUMMARY=1 uv run pytest tests/pytest/test_storefront_agent_eval.py:: -q

We opened the Eval Protocol UI and watched the magic happen. The  rollout UI  in eval-protocol logs showed us the agent's entire thought process: it received the system prompt, planned its response, called the execute_sql tool through the MCP, and generated a final answer, all while the live traffic is going Seeing the full stack come alive with a single command was super fun for me. This "hello world" moment gave us the momentum to start encoding our real product requirements into tests. From Vague Expectations to Concrete Tests

Now for the fun part: turning our project goals into machine-checkable tests that developers can trust. First Acceptance Test: "Find Jazz tracks under $0.99"

We started with a classic user request. Here's how we defined success: • User's Goal:  Get a list of the first 10 Jazz tracks that cost $0.99. • Expected Outcome: The final response must contain 10 specific, correct track names. The list must be formatted in a clean markdown table.

• Scoring Logic: • +0.08 points for each correct track found (for a total of 0.8). • +0.2 points if the output contains a valid markdown table. • Pass Threshold: 0.9  (allowing for minor formatting issues but requiring all tracks).

The agent passed! The rollout UI confirmed that it found all 10 tracks and rendered the table perfectly. This single test set the tone: we could define human-readable outcomes and judge them with deterministic, automated scores. The Iterative Flow

This first success created a powerful development loop. I asked Cursor to add more tests based on our  project.md  file: This is great! Now please add one more example from @project.md and run the test so I can check the rollout in the UI. No need to add the assertion yet. Cursor added new test cases for scenarios like: • Filtering by Media Type:  Find "Rock" music available in "AAC" format. The test asserts that exactly two golden rows are returned and penalizes any extras. • Faceted Search:  Find Jazz tracks between 180-240 seconds long...