Shared Or Dedicated Inference For Embed Rerank

Shared vs. Dedicated AI Inference: Embed & Rerank Skip to content Introducing North 2: AI without compromises Learn more about the updates

Products

Solutions

Resources

Blog

Research

Company

Sign in

Request a demo

Platform North

Enterprise-ready AI for business

Compass

Intelligent search and discovery

Models Command

Generative language models

Transcribe New

Speech recognition model

North Small Translate NEW

Machine translation model

Parse New

Document parsing model

Embed

Semantic representation model

Rerank

Retrieval optimization model

Models Overview

Product Products Overview

Total Cost of AI Ownership

Pricing

Featured North 2: Cohere's latest platform update — with 15+ new features

Deploy Model Vault

Dedicated model inference platform

Private Deployments

On-prem or isolated VPCs

Security

Protect your data at every stage

See deployment options

By Industry Financial Services

Public Sector

Technology

Telecommunications

Energy and Utilities

Healthcare

Manufacturing

Featured Model Vault provides fully-isolated, performant inference with Saas simplicity

Insights Customer Stories

For Developers Developers

Models Overview

Docs

Discord

LLM University

Connect Partners

Events

Webinars

Merch Store

Featured How CoreWeave used Cohere North to transform its customer support in 90 days

Blog

The latest news, launches, and insights

Read more

North 2: Enterprise AI without compromises

Introducing Embed 5—A new family of frontier embedding models

Cohere and Aleph Alpha sign agreement to become the first transatlantic sovereign AI solution

Research Cohere Labs

Cohere’s ML research lab

Explorations Future(s) of Work

How will AI change the way we work?

Aya Models

Multilingual AI at scale

All Papers

Initiatives Research Scholars

Finding the new generation of ML talent

Open Science Community

Championing global, open science

Catalyst Grants

Supporting impactful ML endeavors

Resources Blog

Hugging Face

Events

Featured The future of work debate has an evidence problem

About

Careers

Newsroom

Oct 09, 2026

7 minute read

Shared or dedicated inference for Embed & Rerank

Key takeaways Request shape matters more than request volume alone. A few large embedding requests can consume significantly more compute than thousands of small search queries. RPM alone is not a reliable sizing metric. Traffic patterns determine cost efficiency. Dedicated inference is generally more economical for sustained, predictable workloads, while consumption-based inference is better suited to variable or bursty traffic. Break-even depends on the workload, not just infrastructure pricing. Request size, token volume, and reranking candidate count significantly influence the economic crossover between shared and dedicated inference. Latency and throughput must be considered together. Maximum throughput doesn't necessarily represent usable production capacity. Infrastructure must meet latency requirements while maintaining cost efficiency.

Shared or Dedicated Inference for Embed and Rerank? Start With the Workload Choosing between shared, consumption-based inference and dedicated, provisioned inference is not simply a question of which option has the lower price.

For embedding and reranking workloads, the answer depends heavily on how the application uses the model. A useful way to think about the decision is:

Request Profile - > Traffic Pattern - > Utilization - > Deployment Choice

Understanding these factors can tell you much more than comparing headline prices. Start with the request profile The first question is: What does a typical model request actually look like?

For embedding, request profiles can vary dramatically. Think about an e-commerce search application for a moment. Before a shopper can search for anything, every product in the store has to be turned into something the search system can match against. So the application takes each product description, runs it through the embedding model, and stores the result. That's the indexing job, a big one the first time through and then a steady stream of smaller batches as inventory changes.To get through it efficiently, the application sends products in bulk: say, around 100 descriptions in a single request, each roughly 150 words (~200 tokens). The user is not waiting on instant results, so what matters is finishing the job, not finishing it fast.

Now contrast that with a shopper typing "waterproof trail running shoes" into the search box. Same model, but this time it's one short query and the user is watching a loading spinner until it comes back.

Both scenarios involve embedding workloads, but they optimize for opposite things. Batch processing is about throughput and price-performance - getting the most work done per dollar. While the single query is about responsiveness, and you accept lower price-performance to achieve it.

For the batch processing and search query scenarios, an embeddings profile might look like this: You're not running the same workload twice, you're asking one model to do two very different jobs. Infrastructure tuned for large batches won't necessarily deliver the same response times when serving thousands of small, latency-sensitive requests. Reranking has a different request shape Rerank workloads introduce another crucial variable into the mix: how many candidate documents you're actually asking the model to evaluate.

Let's stick with our product search example. First, your vector search might pull up 50 candidate products that match the query. Then the reranker steps in, it's like having a knowledgeable sales associate who looks at those 50 options and decides which ones truly deserve the top spots.

The model essentially compares the shopper's query against each candidate and ranks them by relevance. Each query-document pair is scored independently, which means the query gets processed once per candidate - so the work scales directly with how many candidates you send.

A typical rerank request looks like this: one short query of around 20 tokens, plus 50 candidate products at roughly 200 tokens each. That's about 11,000 tokens the model works through for a single search.

Simple enough, right? But here's where things get interesting. Suppose you change your strategy from reranking 20 candidates to 50 or 100. On the surface nothing changes for the user- same number of searches, same experience....