microsoft/Power-CAT-Copilot-Studio-Kit

microsoft/Power-CAT-Copilot-Studio-Kit

Language: TypeScript

License: MIT

Stars: 403

Forks: 75

Open issues: 68

Created: 2024-04-25T12:19:36Z

Pushed: 2026-06-10T10:16:03Z

Default branch: main

Fork: no

Archived: no

README:

Power CAT Copilot Studio Kit

The Power CAT Copilot Studio Kit is a comprehensive set of capabilities designed to augment Microsoft Copilot Studio. The kit helps makers develop, govern, and test custom agents, use a large language model to validate AI-generated content, optimize prompts, and track aggregated key performance indicators of their custom agents.

Features

Testing capabilities

Copilot Studio Kit allows makers to configure agents, tests and test sets, and use them to batch test their custom agents. Test runs produce detailed results including latencies, observed responses and run level aggregates. Different test types include response match, attachment match, topic match, multi-turn and generative answers which leverages AI Builder for response analysis.

*Copilot Studio Kit test automation now supports user-defined rubrics for generative answers.*

More information on [testing capabilities](./TESTING_CAPABILITIES.md)

Compliance Hub

Compliance Hub helps organizations define and enforce governance policies for Copilot Studio agents at scale. It continuously evaluates agent configurations collected via Agent Inventory against configurable risk thresholds, automatically creating compliance cases when violations are detected and tracking them through an SLA-driven review lifecycle.

Administrators can configure thresholds, risk levels, SLA timers and enforcement actions (manual review, quarantine, delete), while makers are guided to remediate issues through Teams/Outlook notifications, intake forms, and dashboards showing open cases and compliance posture. This enables makers to innovate freely within Copilot Studio, while ensuring a consistent, auditable level of compliance across all agents.

More information on [Compliance Hub](./COMPLIANCE_HUB.md)

Agent Insights Hub (Preview)

Agent Insights Hub is a comprehensive analytics and monitoring dashboard for Microsoft Copilot Studio agents. It aggregates telemetry from Azure Application Insights, and Conversation Transcripts to provide visibility into agent performance, conversation metrics, topic and tool analytics, error tracking, and credit consumption — all from a single unified interface. The dashboard includes KPI cards with trend indicators, time-series charts, transcript analysis with CSAT scores, and flexible filtering by agent, date range, channel, and data mode. It supports on-demand syncs up to 365 days of historical data, scheduled daily syncs, and Excel export for offline reporting.

More information on Agent Insights Hub

Agent Debugger

Agent Debugger is a diagnostic tool inside the Copilot Studio Kit Admin app that lets administrators load any recorded conversation and inspect every decision the agent made — step by step, with timing, token usage, knowledge sources, arguments, and observations — without leaving the Power Platform.

More information on Agent Debugger

Agent Library (Preview)

The Agent Library is a centralized hub within the Copilot Studio Kit that helps makers discover, install, and manage pre-built agent templates and reusable components. Instead of building agents from scratch, you can browse a curated catalog of ready-to-use templates (Custom Agent and Declarative Agent) and components, install them directly into your environment, and customize them to fit your business needs.

More information on Agent Library

Component Library (Preview)

The Component Library is a collection of ready-to-use, pre-built components for Microsoft Copilot Studio — packaged as component collections so you can import only the ones you need. The new app lets you easily browse and add the component to your environment. Add the component to your agent like you would with any component collection. Use agent instructions to control when and how each component is invoked — no custom development required.

More information on Component Library

Rubrics refinement

Rubrics Refinement is a powerful capability in Copilot Studio Kit that enables you to create, test, and iteratively improve reusable evaluation standards (rubrics) for AI-generated responses. This feature helps ensure that AI grading of your agent's responses aligns with human judgment and organizational quality standards.

More information on [Rubrics Refinement](./RubricRefinement/01-rubrics-refinement-overview.md)

Conversation KPI

Conversation KPI are designed to help makers track and analyze the performance of their custom agents. This feature complements the existing analytics built-in the Copilot Studio and simplifies the process of understanding conversation outcomes by providing aggregated data in Dataverse rather than requiring you to analyze the complex conversation transcripts.

More information on [conversation KPI](./CONVERSATION_KPI.md)

SharePoint synchronization

SharePoint synchronization allows makers to configure periodical selective content synchronization from SharePoint locations to custom agent knowledge base as files.

More information on [SharePoint synchronization](./FILE_SYNCHRONIZATION.md)

Prompt Advisor

Prompt Advisor allows makers to develop effective prompts while learning useful prompt engineering skills. Prompts entered in the advisor tool will be analyzed and receive a confidence evaluation with detailed reasoning for the score. Advisor also provides a list of suggested refined prompts implementing various prompt techniques. Makers can select from these optimized prompts to iteratively refine and improve their input.

More information on [Prompt Advisor](./PROMPT_ADVISOR.md)

Webchat Playground

Webchat Playground simplifies customizing the appearance and behavior of the copilot agent webchat, including colors, fonts, thumbnails and initials. Easy to use UI allows makers to define the look and feel of their webchat, and HTML is generated with the…