microsoft/rag-experiment-accelerator

microsoft/rag-experiment-accelerator

Description: The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.

Language: Python

License: NOASSERTION

Stars: 309

Forks: 109

Open issues: 156

Created: 2023-09-25T16:47:36Z

Pushed: 2026-06-15T00:24:22Z

Default branch: development

Fork: no

Archived: no

README:

RAG Experiment Accelerator

Overview

The RAG Experiment Accelerator is a versatile tool that helps you conduct experiments and evaluations using Azure AI Search and RAG pattern. This document provides a comprehensive guide that covers everything you need to know about this tool, such as its purpose, features, installation, usage, and more.

Purpose

The main goal of the RAG Experiment Accelerator is to make it easier and faster to run experiments and evaluations of search queries and quality of response from OpenAI. This tool is useful for researchers, data scientists, and developers who want to:

• Test the performance of different Search and OpenAI related hyperparameters.
• Compare the effectiveness of various search strategies.
• Fine-tune and optimize parameters.
• Find the best combination of hyperparameters.
• Generate detailed reports and visualizations from experiment results.

Latest changes

18 March 2024: Content sampling has been added. This functionality will allow the dataset to be sampled by a specified percentage. The data is clustered by content and then the sample percentage is taken across each cluster to attempt even distribution of the sampled data.

This is done to ensure representative results in the sample that one would get across the entire dataset.

Note: It is recommended to rebuild your environment if you have used this tool before due to new dependencies.

Features

The RAG Experiment Accelerator is config driven and offers a rich set of features to support its purpose:

1. Experiment Setup: You can define and configure experiments by specifying a range of search engine parameters, search types, query sets, and evaluation metrics.

1. Integration: It integrates seamlessly with Azure AI Search, Azure Machine Learning, MLFlow and Azure OpenAI.

1. Rich Search Index: It creates multiple search indexes based on hyperparameter configurations available in the config file.

1. Multiple Document Loaders: The tool supports multiple document loaders, including loading via Azure Document Intelligence and basic LangChain loaders. This gives you the flexibility to experiment with different extraction methods and evaluate their effectiveness.

1. Custom Document Intelligence Loader: When selecting the 'prebuilt-layout' API model for Document Intelligence, the tool utilizes a custom Document Intelligence loader to load the data. This custom loader supports formatting of tables with column headers into key-value pairs (to enhance readability for the LLM), excludes irrelevant parts of the file for the LLM (such as page numbers and footers), removes recurring patterns in the file using regex, and more. Since each table row is transformed into a text line, to avoid breaking a row in the middle, chunking is done recursively by paragraph and line. The custom loader resorts to the simpler 'prebuilt-layout' API model as a fallback when the 'prebuilt-layout' fails. Any other API model will utilize LangChain's implementation, which returns the raw response from Document Intelligence's API.

1. Query Generation: The tool can generate a variety of diverse and customizable query sets, which can be tailored for specific experimentation needs.

1. Multiple Search Types: It supports multiple search types, including pure text, pure vector, cross-vector, multi-vector, hybrid, and more. This gives you the ability to conduct comprehensive analysis on search capabilities and results.

1. Sub-Querying: The pattern evaluates the user query and if it finds it complex enough, it breaks it down into smaller sub-queries to generate relevant context.

1. Re-Ranking: The query responses from Azure AI Search are re-evaluated using LLM and ranked according to the relevance between the query and the context.

1. Metrics and Evaluation: It supports end-to-end metrics comparing the generated answers (actual) against the ground-truth answers (expected), including distance-based, cosine and semantic similarity metrics. It also includes component-based metrics to assess retrieval and generation performance using LLMs as judges, such as context recall or answer relevance, as well as retrieval metrics to assess search results (e.g. MAP@k).

1. Report Generation: The RAG Experiment Accelerator automates the process of report generation, complete with visualizations that make it easy to analyze and share experiment findings.

1. Multi-Lingual: The tool supports language analyzers for linguistic support on individual languages and specialized (language-agnostic) analyzers for user-defined patterns on search indexes. For more information, see Types of Analyzers.

1. Sampling: If you have a large dataset and/or want to speed up the experimentation, a sampling process is available to create a small but representative sample of the data for the percentage specified. The data will be clustered by content and a percentage of each cluster will be selected as part of the sample. Results obtained should be roughly indicative of the full dataset within a ~10% margin. Once an approach has been identified, running on the full dataset is recommended for accurate results.

Products used

• Azure AI Search Service (Note: Semantic Search is available in Azure AI Search, at Basic tier or higher.)
• Azure OpenAI Service or access to the OpenAI API
• Azure Machine Learning Resources

Compute setup

At the moment, the RAG Experiment Accelerator can be run locally leveraging...