microsoft/eval-guide

microsoft/eval-guide

Description: A plugin for AI agent evaluation. Plan evals, generate test cases, interpret results for Copilot Studio agents. Grounded in Microsoft's Eval Scenario Library & Triage Playbook.

Language: HTML

License: MIT

Stars: 133

Forks: 21

Open issues: 1

Created: 2026-03-30T17:24:37Z

Pushed: 2026-06-24T19:13:52Z

Default branch: main

Fork: no

Archived: no

README:

eval-guide

AI agent evaluation toolkit for Copilot Studio. Plan evals, generate test cases, interpret results, and triage failures — from Claude Code or GitHub Copilot.

Grounded in Microsoft's Eval Scenario Library, Triage & Improvement Playbook, Common Evaluation Approaches, and MS Learn agent evaluation documentation.

Install

Claude Code

claude plugin marketplace add microsoft/eval-guide
claude plugin install eval-guide@eval-guide

GitHub Copilot

npx skills add microsoft/eval-guide

Skills

| Skill | Command | What it does | |-------|---------|-------------| | Eval Guide | /eval-guide | Full eval lifecycle — discover, plan, generate, run, interpret. Start here. | | Eval Suite Planner | /eval-suite-planner | Populated Eval Suite Template workbook plus an interactive HTML review page for eval sets, methods, gates, human inputs, and grader-validation notes | | Eval Generator | /eval-generator | Test cases for single-response and conversation (multi-turn) evaluation modes | | Eval Result Interpreter | /eval-result-interpreter | SHIP / ITERATE / BLOCK verdict with root cause classification | | Eval Triage & Improvement | /eval-triage-and-improvement | Interactive diagnosis and remediation for failing evals | | Eval FAQ | /eval-faq | Methodology questions answered from Microsoft's eval ecosystem |

Quick start

> /eval-guide

Tell me about your agent — what does it do, who uses it, and what does "good" look like?

Works the same in both Claude Code and GitHub Copilot.

The toolkit walks you through five operational stages over Microsoft's ***Practical Guidance on Agent Evaluation* — 10-step playbook** (the canonical methodology spine, in skills/eval-guide/playbook.md):

| Stage | What happens | Playbook steps | Works without a running agent? | |-------|-------------|----------------|-------------------------------| | 0. Discover | Articulate what the agent does, what success looks like, the eval objective, the agent's risk tier (5 factors), and the owner | Step 1 | Yes | | 1. Plan | Scope eval depth by agent architecture; plan capability vs trust & safety eval sets; set pass-rate targets and hard/soft gates; specify human inputs + source→ground-truth map | Steps 1, 4, 5 | Yes | | 2. Generate & Baseline | Produce capability and trust & safety test-case CSVs (single-response) or conversation blueprints (multi-turn) importable into Copilot Studio; design the regression partition | Steps 2, 3, 8 (design) | Yes | | 3. Run | Execute the baseline against a live agent | Step 6 | Needs running agent | | 4. Interpret & Improve | Triage results, classify each failure (eval-setup vs agent-quality), gate-based verdict, design the optimization loop, flag reusable assets | Steps 7, 9, 10 | Needs eval results |

Stages 0-2 work from just an agent description — no running agent required.

Interactive dashboard review

Each stage generates an interactive HTML dashboard served locally in your browser. You review, edit inline, and confirm before the AI proceeds — no more back-and-forth in chat to fix test cases.

Stage complete → Dashboard opens → You review & edit → Confirm → Final artifacts generated

| Stage | What you review in the dashboard | What you can edit | |---|---|---| | 0. Discover | Agent Vision (purpose, users, knowledge, capabilities, boundaries, success criteria) | All fields inline, add/remove list items | | 1. Plan | Populated Eval Suite Template workbook plus HTML review page | Edit workbook cells without changing template structure; use the page to review summary, filters, TBDs, and checklist | | 2. Generate | Test cases per eval set | Edit expected responses, questions, methods, add/remove cases | | 4. Interpret | Verdict, failure triage, root causes, actions | Reclassify root causes, add comments |

Final deliverables (.docx reports, .csv test sets) are only generated after you confirm via the dashboard.

The dashboard is a standalone HTML file generated by skills/eval-guide/dashboard/serve.py (zero dependencies) and opened directly in your browser — no server required. Feedback auto-saves as you edit via localStorage — if the browser closes, your work is preserved.

Architecture-aware eval scoping

The toolkit automatically scopes evaluation depth based on your agent's architecture:

| Architecture | What gets tested | |---|---| | Prompt-level (simple Q&A, no knowledge sources) | Response quality, tone, boundaries, refusal behavior | | RAG / Knowledge-grounded (has knowledge sources, no tools) | All of the above + retrieval accuracy, grounding, hallucination prevention | | Agentic (multi-step, tool use, orchestration) | All of the above + tool selection, action correctness, error recovery, task completion |

A simple FAQ bot doesn't need tool-routing tests. A multi-step workflow agent does. The toolkit handles this so you test what actually matters.

Single-response and conversation evaluation

The eval generator supports both modes:

• Single-response — one input, one output. Produces a CSV importable directly into Copilot Studio. Supports all 7 test methods (General Quality, Compare Meaning, Keyword Match, Text Similarity, Exact Match, Capability Use, Custom).
• Conversation (multi-turn) — multi-turn dialogues for agents that handle context-dependent, multi-step tasks. Produces a structured blueprint for creating conversation test sets in Copilot Studio. Supports General Quality, Keyword Match, Capability Use, and Custom.

The skill detects which mode fits your agent and recommends accordingly.

Test data generation strategies

The planner recommends the right...