anthropics/anthropic-retrieval-demo

anthropics/anthropic-retrieval-demo

Description: Lightweight demo using the Anthropic Python SDK to experiment with Claude's Search and Retrieval capabilities over a variety of knowledge bases (Elasticsearch, vector databases, web search, and Wikipedia).

Language: Python

License: MIT

Stars: 195

Forks: 50

Open issues: 12

Created: 2023-09-08T23:41:15Z

Pushed: 2024-06-30T19:18:19Z

Default branch: main

Fork: no

Archived: yes

README:

Claude Search and Retrieval Demo [Experimental]

Introduction

Lightweight demo using the Anthropic Python SDK to experiment with Claude's Search and Retrieval capabilities over a variety of knowledge bases (Elasticsearch, vector databases, web search, and Wikipedia). In this demo, we explore an alternative to tradional retrieval-augmented generation (RAG) techniques.

The repository contains the following directories:

|Directory | Description| |--- | ---| |claude_retriever | Contains the core search and retrieval logic using Claude's API.| |examples | Provides example notebooks and scripts showing how to use claude_retriever with various search tools.| |tests | Includes unit and integration tests for the embedders, utils, and search tools.|

Table of Contents

• [Setup](#setup)
• [General Environment Variables](#General-Environment-Variables)
• [How it works](#usage)
• [Explanation of core methods](#explanation-of-core-methods)
• [A peek under the hood](#retrieval-under-the-hood)
• [Examples](#examples)
• [Wikipedia Search](#using-a-search-api-to-access-wikipedia)
• [Vector Database Search](#setting-up-and-using-an-embedding-database)
• [Web Search (Brave)](#searching-the-internet-with-bravesearchtool)
• [Enterprise Search (Elasticsearch)](#using-elasticsearch-as-a-search-tool)

Setup

This retrieval demo uses Python 3.10. Install it if you don't already have it.

Clone the repository:

git clone https://github.com/anthropic/claude-retriever-demo.git

Navigate into the demo directory:

cd claude-retriever-demo

It's recommended to use a virtual environment. Create and activate one:

python3 -m venv venv
source venv/bin/activate

Install the demo:

pip install -e .

For conda or other virtual environment managers, once you've cloned the repo, create a new environment:

conda create --name retrieval
conda activate retrieval

Install requirements:

conda install pip
pip install -r requirements.txt

General Environment Variables

You'll need to add some environment variables, at minimum:

|Environment Variable | Description| |-|-| |ANTHROPIC_API_KEY | API key for Anthropic's Claude. You can apply for an API key here.|

Once set, refer to the example notebooks in [/examples](/examples) to start testing retrieval workflows.

Usage

At a high level, the Retrieval Demo works as follows: 1. Set up a SearchTool object that can take natural queries and return formatted search results. 2. Initialize a ClientWithRetrieval object, which inherits from the Client object in the base Anthropic SDK. It accepts a SearchTool object. 3. Perform retrieval via the completion_with_retrieval method which is similar the base completion method, but Claude will issue queries and use the SearchTool to better solve the task given to it. This is the quickest way to experiment with retrieval.

We also offer two methods that are used within completion_with_retrieval that can also be used standalone:

• retrieve: The retrieve method allows you to make a single call to get Claude to gather relevant information for a given question task. This provides greater steerability of the RAG pipeline by allowing for you to use the search results downstream and apply additional post-processing.
• answer_with_results: The answer_with_results method performs the traditional retrieval-augmented generation step and uses Claude to provide an answer to a query given search results as context.

Explanation of core methods

answer_with_results()

This method comprises the traditional synthesis step in most RAG pipelines today. Here we provide Claude the search results and ask it to synthesize an answer to the user's question.

!answer_with_results diagram It works through the following steps: 1. The user's question and the search results are passed into the method. 2. If the format_search_results parameter is set to True then we reformat the search results into the format that Claude has been finetuned to expect. 3. The formatted search results plus the original question are passed to Claude to generate a completion. 4. Claude reads over the search results to extract relevant information and synthesizes an answer to the user's question. 5. The Claude-generated answer is returned.

To integrate Claude into your traditional RAG pipeline, you can call the answer_with_results() method by itself. Set the format_search_results parameter to True and pass in a list of your raw search results (in the form of a list[str]) to the method.

retrieve()

The traditional RAG pipeline only uses LLMs for answering after search results are gathered from the knowledge base. The retrieve() method leverages Claude's capabilities earlier in the process.

retrieve() allows Claude to iteratively search the knowledge base to gather relevant information until it decides enough has been collected to answer the question. Through this approach, we use Claude to assist in gathering more relevant information than would be possible if we only relied on the inital user question as the search query.

!retrieve diagram

Here is the flow:

1. The original query is passed to Claude. 2. Claude generates natural language search queries based on the original query. 3. The searches are passed to the search tool. Documents are returned. 4. Claude receives the search results. 5. Claude evaluates if enough information has been gathered. 6. Claude continues searching up to max_searches_to_try. 7. Once done, the final set of search results are cleaned (deduplicated, formatted) and returned.

It's important to note that retrieve() returns the search results rather than a final answer. This allows you to do further processing on the results downstream. The modular nature provides more control over the search process.

Search tool

A…