7 Long Context Trends In The Enterprise

Long Context Trends in the Enterprise: 7 Common Use Cases From the Field | AI21

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

In 50 meetings with enterprise clients over the last three months alone, the use cases they describe wanting to build are what we in the field call long context use cases: Tasks that are data-heavy and require a model to be able to ingest a large amount of information in order to produce an accurate and thorough response.

This market trend is confirmed by the increasingly longer context windows we’re seeing with each subsequent language model release, with some models starting to push into the millions of tokens—a huge leap from the 4K or 8K token counts of barely two years ago.

A big demand for this rise in context window length comes from the business world, where long context is instrumental in scaling applications to support enterprise workflows. It’s the difference between a developer feeding a model the file they are working on versus their entire organizational codebase. The scale—and the resulting intelligence—is incomparable.

However, with context windows always listed in tokens, the practical difference between a 32K and 256K model, for example, is not always clear. Companies want to know: How many financial reports does that add up to? How many interactions in a customer support chat? And, of course, what tangible value does the ability to fit so much context have on model output?

In this blog, I want to spend some time explaining the value of long context, before sharing the top 7 long context use cases I see across the financial services, health care and life sciences industries, as well as cross-industry customer support use cases— plus a bonus prompt example and cost and latency breakdown for an earnings call summarization use case. In each use case, I’ll share how long context solved a core business challenge, bridging the journey from ideation to production.

Context window 101

Context window is the total input and output data that a model can ingest at any given moment. It’s a moving target, though, so as soon as the context window fills up, the model will start losing access to the data that came at the start.

People often use the metaphor of being in a (very) long conversation, and starting to get hazy about how or where you started off once you’re an hour in.

Because of the finite nature of context windows, the enterprises I speak with care a lot about having a model whose long context can adequately support the information they need the model to ingest and process. In particular, long context offers enterprises greater customization and better quality output, while optimizing for cost.

Model output customized to your organization

During pre-training, an LLM is exposed to vast amounts of general data that instill it with some basic context and also help it develop strong reasoning skills. Yet to successfully answer questions that are highly particular to a company and their operations, the model will need a bit more information: your unique organizational nomenclature, brand guidelines, customer service protocol, refund policies, and more.

This is information you don’t want the model reasoning from its training data, and the model needs the ‘room’ in its context window to fit this information, along with the prompt.

More accurate and contextualized output

Long context allows a model to apply good research skills. Just as someone who has read just one article about a topic will not give answers that are as satisfactory and in-depth as someone who has read a whole book on the same topic, a model with a smaller window will always be limited in the number of documents it can process to generate an answer. While workarounds, such as summarization mechanisms, exist for breaking long documents into shorter chunks, these strategies are less reliable than models trained to handle long context from the outset.

For question answering or summarization tasks on lengthy documents or a large number of documents, a model that can take in more context will have higher success producing an answer that is both accurate and also makes sense in its broader context.

Optimized costs

As many enterprises with RAG systems know, the costs of these solutions can quickly balloon when reviewing and synthesizing large quantities of information. Limiting the amount of snippets retrieved could be one workaround—but it risks missing out on critical information or context. Not to mention, for industries who rely on documents that regularly reach 100+ pages, such as finance, quickly filling up a context window is inevitable.

For those cases, using a cost-effective, high-performing long context model alongside your RAG system is essential for keeping costs in check. Not only is Jamba-Instruct one of the few models on the market to maintain impressive performance on long context use tasks across its entire 256K context window, it also offers this massive context window at one of the best price points on the market, making it an attractive option for companies who want both a reliable and cost-effective RAG solution.

Top long context tasks for the enterprise

A long context model, like AI21 Labs’ Jamba-Instruct, which has an effective context window of 256K , can help enterprises with the following:

Multi-document analysis: Summarize or compare across multiple documents at once to identify key points and insights.

Multi-document question answering: Query multiple documents, records, or policies in a database.

Organizational search assistant: Improve the retrieval stage of a RAG system for organizational data, resulting in higher quality answers.

Risk mitigation : Inject a prompt with complex and detailed instructions to guide its response, especially important in highly-regulated industries like finance and healthcare.

In the following paragraphs, I’ll walk through some of these tasks by industry to show how a long context model, like AI21 Labs’ Jamba-Instruct model—with a 256K context window—is necessary for high quality and reliable output.

Finance

Use case: Extracting business insights from earnings calls

As part of gathering and synthesizing intelligence about a given company’s performance, investment analysts will comb through their quarterly earnings calls, using the updates shared there to make recommendations to portfolio or investment managers. The transcripts of these calls can be around 19...