Inside DigitalOcean Simulations: how user simulation, execution, and validation work

Inside DigitalOcean Simulations: how user simulation, execution, and validation work | DigitalOcean

© 2026 DigitalOcean, LLC. Sitemap .

Engineering

Inside DigitalOcean Simulations: how user simulation, execution, and validation work

By Venkat Ranabothu

Published: October 29, 2026 15 min read

Evaluations .

Here’s how to run Evaluations:

Find the agents . We compare two configurations of the same agent.

The baseline’s instructions say “answer Python questions concisely.”

The candidates add: “Look up current release information with web fetch instead of answering from memory. When there are options, present them and let the user choose. For upgrades, cover every release between the current and target version, and check the user’s code before giving a verdict.”

doctl harness-runtime config list

Pick the metrics . Use the UUIDs of the conversational metrics you want to measure:

doctl evaluation metric list --format UUID,Name,Category

Create the scenario set . Upload the scenario:

doctl evaluation scenario-set create \ --name python-upgrade \ --file ./scenarios.jsonl

Or give a goal description and let the generator model write the scenarios:

doctl evaluation scenario-set generate \ --name python-upgrade-gen \ --goal-description "A developer asks a coding agent which Python version to upgrade a service to and what will break. The agent should look up current release information, let the developer choose from the supported versions, cover breaking changes in every release between the current and chosen version, and check the developer's code for anything affected." \ --num-scenarios 5

Generation runs in the background. Wait for the status to go from GENERATING to READY:

doctl evaluation scenario-set get \ --format UUID,Name,Status,ScenarioCount

Then check what you are about to test. Each description should explain the situation, and each stopping criterion should describe what the agent must accomplish:

doctl evaluation scenario-set list-scenarios \ --format Name,Description,UserPersona,MaxTurns

Full scenarios, including stopping criteria

doctl evaluation scenario-set list-scenarios -o json

Run the baseline and the candidate. The only difference between the two commands is the agent UUID:

doctl evaluation simulation-run create \ --name python-upgrade-baseline \ --scenario-set-uuid \ --agent-uuid \ --metric-uuids , , , \ --star-metric-uuid \ --star-metric-success-threshold 80 \ --max-turns 6 \ --exploration-budget 1

doctl evaluation simulation-run create \ --name python-upgrade-candidate \ --scenario-set-uuid \ --agent-uuid \ --metric-uuids , , , \ --star-metric-uuid \ --star-metric-success-threshold 80 \ --max-turns 6 \ --exploration-budget 1

--max-turns and --exploration-budget override the values on each scenario.

To test a model or Inference Router instead of a managed agent, use the API for now:

curl -X POST "https://api.digitalocean.com/v2/gen-ai/simulation_runs" \ -H "Authorization: Bearer $DIGITALOCEAN_TOKEN " \ -H "Content-Type: application/json" \ -d '{ "name": "python-upgrade-model", "scenario_set_uuid": "", "candidate_model_uuid": "", "candidate_model_source": "CANDIDATE_MODEL_SOURCE_SERVERLESS", "model_access_key": "", "evaluation_config": { "metric_uuids": [""] } }'

Monitor the run . It moves through Pending, Running, and Evaluating, and ends Succeeded, Partially successful, Failed, or Cancelled:

doctl evaluation simulation-run get \ --format Name,Status,TotalEpisodes,EpisodesFinished

The JSON output also carries the linked evaluation run UUID

doctl evaluation simulation-run get -o json

Read the trajectories . List the episodes, then open any trajectory to see what the agent did:

doctl evaluation simulation-run list-episodes \ --format UUID,ScenarioUUID,EpisodeIndex,Status,DurationSec

doctl evaluation simulation-run get-trajectory -o json

Download link for the full trajectory file

doctl evaluation simulation-run get-trajectory-url

Each trajectory has the messages, tool calls, turn count, and per-metric scores with the judge’s reasoning. This is where you can see whether the baseline fetched anything in turn 1, and which release notes it read in turn 2.

Read the evaluation results . Each simulation run has a linked evaluation run, with scores grouped by scenario and then by episode:

doctl eva