Fine Tuning Your Own Embeddings Model

Fine-Tune Your Own Embedding Model from an LLM — for the Price of a Coffee

Kimi K3 on Fireworks: Frontier Intelligence You Can Own

Blog

Fine Tuning Your Own Embeddings Model Fine-Tune Your Own Embedding Model for the Price of a Coffee

PUBLISHED 7/29/2026

Table of Contents Why Fine-Tune an LLM into an Embedding Model? How Embedding Fine-Tuning Works: In-Batch Negatives and InfoNCE Experimental Results

Experiment 1: Legal Citation Retrieval (LegalBench)

Experiment 2: Clinical Trial Matching (TREC Clinical Trials)

Experiment 3: EU Case-Law Citation Retrieval (LegalPincite) Code Example: Train on Your Private Data What Else We Found

When Fine-tuning Didn't Move The Needle Getting Started

Three Fireworks Training Modes: Flexibility vs. Efficiency

Interested in Fireworks Training?

Table of Contents

TL;DR

Retrieval-augmented generation (RAG), semantic search, reranking, recommendation, intention classification, deduplication, clustering — an enormous share of production AI boils down to one primitive: turning text into a vector so that related things land close together and unrelated things land far apart. The quality of that vector space is the ceiling on every system built on top of it. A great reranker cannot rescue a retriever that never surfaced the right document; a RAG pipeline cannot cite a case it failed to retrieve. So the question becomes: how do you get the best possible embeddings for your domain? In this post we show a practical, low cost recipe (", "positive": ""} . You don't need to supply negatives — in-batch negatives are generated automatically. Then run the contrastive fine-tune through the Fireworks Training SDK. The trainer is provisioned for you; the loop streams batches, computes the bidirectional InfoNCE loss, and promotes the final checkpoint to a model you can deploy: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 import os import training . recipes . embedding_loop as embedding_loop from training . utils import TrainerConfig

config = embedding_loop . Config ( log_path = "./embedding_logs" ,

Start from a strong pretrained embedding LLM — do NOT train from scratch.

base_model = "accounts/fireworks/models/qwen3-embedding-8b-ft-base" , tokenizer_model = "Qwen/Qwen3-Embedding-8B" ,

Your private (query, positive) pairs.

dataset = "my_domain_pairs.jsonl" ,

Contrastive objective: bidirectional InfoNCE over in-batch negatives.

output_mode = "embedding" , # "contrastive_loss" runs it server-side in one round trip temperature = 0.02 , # sharpen the softmax batch_size = 64 , # 64 -> 63 in-batch negatives per query learning_rate = 1e-5 , epochs = 3 , # ~150 steps is plenty for most datasets

A query-side instruction tailored to YOUR task (Qwen3 convention).

query_instruction = "Given a legal question, retrieve court opinions that should be cited in the answer" ,

lora_rank=32 for a cheap, swappable adapter; 0 = full-parameter.

lora_rank = 0 ,

Promote the trained checkpoint to a deployable model.

output_model_id = "qwen3-embedding-8b-my-domain" ,

trainer = TrainerConfig ( ) , # infra (GPU/region) resolved by the platform )

metrics = embedding_loop . main ( config ) print ( "Final metrics:" , metrics )

Once training finishes, your model serves through the standard embeddings endpoint — same API as the base model, now specialized to your domain: 1 2 3 4 5 6 7 8 9 from fireworks . client import Fireworks

client = Fireworks ( api_key = os . environ [ "FIREWORKS_API_KEY" ] )

resp = client . embeddings . create ( model = "accounts//models/qwen3-embedding-8b-my-domain" , input = [ "In Florida, what authority governs frivolous pro se prisoner petitions?" ] , ) query_vec = resp . data [ 0 ] . embedding # L2-normalized; dot product = cosine similarity

Or call the /v1/embeddings endpoint directly (the response is OpenAI-compatible): 1 2 3 4 5 6 7 curl -s https://api.fireworks.ai/inference/v1/embeddings \ -H "Authorization: Bearer $FIREWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "accounts//models/qwen3-embedding-8b-my-domain", "input": ["In Florida, what authority governs frivolous pro se prisoner petitions?"] }'

Index your corpus once with the same model, and retrieval is a nearest-neighbor lookup over these vectors. What Else We Found

Beyond the two headline results, we ran a series of sweeps. The details are out of scope for this post, but the takeaways are worth stating: • Fine-tuning's gain tracks the base model's headroom. Where the base is already strong, gains are small; where it struggles, they are large. We saw the same pattern on three more domains — code search ( CoSQA ), developer Q&A ( StackOverflow-QA ), and e-commerce product search ( Amazon ESCI ): a clear lift on ESCI, a small one on the already-near-ceiling StackOverflow-QA, and essentially flat o