openai/openai-knowledge-retrieval

openai/openai-knowledge-retrieval

Description: OS repo for Knowledge Retrieval starter kit

Language: Python

License: MIT

Stars: 72

Forks: 28

Open issues: 0

Created: 2025-10-31T21:31:22Z

Pushed: 2026-01-13T20:41:07Z

Default branch: main

Fork: no

Archived: no

README:

OpenAI Knowledge Retrieval

Config-first RAG starter kit that pairs OpenAI File Search, Chatkit, and Evals with a pluggable ingestion, retrieval, and evaluation toolkit.


Quickstart

Launch a working knowledge retrieval app in a few commands:

# 1. Backend environment
python3 -m venv .venv
source .venv/bin/activate
make dev # or: make install

# 2. Credentials
cp .env.example .env
echo "OPENAI_API_KEY=sk-..." >> .env

# 3. Update configuration
# Edit the configuration in configs/default.openai.yaml to point to your documents.

# 4. Ingest corpus
make ingest

# 5. Run backend + frontend (in separate terminals)
make run-app-backend

# 6. Run frontend (in another terminal)
make run-app-frontend

# 7. Open to chat with the assistant. All answers are grounded with citations that link back to the ingested documents.

# 8. Evals: Generate synthetic data for evaluation and upload to OpenAI Evals
make eval

Features

• YAML-first configuration – define ingestion, retrieval, and synthesis behavior without editing

code.

• Multiple vector stores – ship with OpenAI File Search or go custom with a local Qdrant DB.
• Typed pipelines – ingestion, retrieval, and synthesis flows built with Pydantic models and

Typer/FastAPI services.

• ChatKit UI – Leverage Chatkit components for interacting with the knowledge base.
• Evaluation harness – Generate synthetic data and run evals with OpenAI Evals or local grading.

Repository Structure

• cli/ – Typer CLI (rag) entrypoint and config parsing.
• ingestion/ – loaders, chunkers, preprocessors, and orchestrated pipelines.
• retrieval/ – query expansion, filtering, reranking, and response assembly.
• stores/ – adapters for OpenAI File Search and custom vector stores (Qdrant example included).
• app/backend/ – FastAPI application that powers the chat API.
• app/frontend/ – Vite + React UI using Tailwind CSS and ChatKit components.
• configs/ – starter default YAMLs you can copy and customize.
• evals/ – evaluation harness, dataset generation, and reporters.
• templates/ – additional templates for different vector stores, chunking strategies, and retrieval methods.
• prompts/ – prompts for the different aspects of the knowledge retrieval workflow (query, expansion, reranking, local judging, etc.)

Pipeline Setup

To do a more custom implementation beyond the quickstart, you can follow the steps below.

1. Prerequisites

• Python 3.10 or later
• Node.js 18.18 or later (for the optional web UI)
• An OpenAI API key (OPENAI_API_KEY) with access to File Search and Evals

2. Install backend dependencies

python3 -m venv .venv
source .venv/bin/activate
make dev # or: make install

3. Configure credentials

Copy .env.example to .env and fill in the values that apply to your environment. At minimum set OPENAI_API_KEY. Optional variables:

OPENAI_ORG_ID=...
OPENAI_PROJECT_ID=...
VECTOR_STORE_ID=vs_...

4. Create a configuration

Use the CLI to copy a YAML config template into configs/ (replace my with a project name):

rag init --template openai > configs/my.openai.yaml

Templates include openai and custom-qdrant, along with different options for chunking data. Adjust data.paths to point at the documents you want to ingest. Example datasets live in data/example_data/.

Add your config to the RAG_CONFIG environment variable.

5. (Optional) Spin up your qdrant vector store in docker

make qdrant-up

This will start a local Qdrant instance on port 6333.

6. Ingest your knowledge base

rag ingest --config configs/my.openai.yaml

The CLI will create or reuse a vector store, upload documents, and print the resulting vector_store_id. Persist that value in your .env.

Optional: set RAG_SKIP_PREFLIGHT=1 to skip preflight de-dupe listings when your org has a large number of files. This starts uploads immediately, but can allow duplicate uploads because de-dupe is bypassed.

7. Launch the experience

Start the FastAPI backend (reads RAG_CONFIG, defaulting to configs/default.openai.yaml):

make run-app-backend

Start the web UI in another terminal:

make run-app-frontend

The UI runs at by default and communicates with the backend at .

8. Evaluate retrieval quality

rag eval --config configs/my.openai.yaml

The harness can either use an existing dataset (evals.mode: user) or synthesize one (evals.mode: auto) When evals.openai_evals.enabled: true, results are mirrored to the OpenAI Evals dashboard.

Configuration & Customization

Every aspect of the pipeline is driven by YAML. Use rag init --template to scaffold a config, then customize the sections described below.

Templates

• openai – Default template backed by OpenAI File Search. Works out of the box with the supplied

example dataset.

• custom-qdrant – Connects to a locally hosted Qdrant instance. Pair with make qdrant-up to start

Docker services automatically.

Add --chunking when scaffolding to pick a predefined chunking approach (see next section).

Chunking strategies

Chunking is controlled under chunking in the config:

• recursive – Recursively backs off from headings to paragraphs, sentences, then tokens to hit the

target window.

• heading – Uses heading hierarchy (#, ALL CAPS, numbered headings) to keep coherent sections.
• hybrid – Falls back to recursive when headings are missing (default).
• xml_aware – Purpose-built for XML manuals; splits on semantic tags and converts tables to

Markdown.

• custom – Load a custom chunker via custom_chunker.{module_path,class_name,init_args}.

Common parameters:

• target_token_range: [min, max] – Token window to aim for before chunking stops.
• overlap_tokens – Overlap between adjacent chunks to maintain context.
• rules – Strategy-specific options (heading detection hints, XML tag boundaries, etc.).

Vector store configuration

Select your store with vector_store.backend:

• openai_file_search – Managed vector store that handles uploads, dedupe, and metadata.…