From Data Chaos To Clarity Unlock Enterprise Ai Value

North Mini Code. Cohere's first model for developers.

Learn more

Oct 07, 2024

6 minutes read

From data chaos to clarity: Unlock enterprise AI value

To optimize AI solutions, companies need a tailored data strategy that addresses common challenges such as quality, connectivity, and scaling.

!Blog Post Featured Image

Updated:

Generative AI (GenAI) systems have the potential to transform company operations, but they’re only as good as the data they can access and use.

Therein lies one of the biggest challenges for enterprises as they seek to harness the power of AI: How to effectively connect to and retrieve data from sources that often vary extensively in quality, format, size, structure, and even language.

To be sure, connecting to reliable, consistent, timely data has long been crucial to getting the most out of enterprise software applications. But to reap the benefits of advanced enterprise AI systems like retrieval-augmented generation (RAG), companies need to raise the bar. The adage “garbage in, garbage out” becomes truer than ever.

The exact approach to tackling data challenges will vary widely depending on a company’s industry and size, and their overall goals for AI adoption. The first step, therefore, is always to establish a clear idea of the use cases for RAG systems.

Some companies, such as those in financial or professional services, might predominantly need access to structured data such as client-specific information from CRM systems, financial and market records, and regulatory compliance data. Others might need to retrieve more unstructured data, such as emails and social media posts, for which language generation is more important than analysis.

With a clear understanding of the data requirements, enterprises can start to tackle some common data challenges. Here, we identify the top three challenges and how solutions like embeddings, data connectors, and reranking can help ease many problems. Let’s dive in.

Addressing poor data quality

Poor data quality has become the primary factor limiting AI adoption, according to consulting firm Forrester. Machine learning models require good data quality for effective training. Generative models have similar requirements – not only during their training, but also when they interpret and synthesize data to generate responses.

In most applications of RAG, a search system retrieves relevant data (textual, tabular, etc.) based on a user query. The quality of the data available directly impacts its searchability and also how well the generative model is able to interpret and build responses based on it. Noisy data – for example, content sources that contain multiple mistakes or gaps – is more likely to lead to inaccurate or irrelevant results, reducing the effectiveness of a system and undermining trust in it.

The issue of data quality can be tackled at different levels. Before connecting data with an AI engine, companies can undertake preprocessing, or cleaning, to address any gaps, errors and inconsistencies in their documentation. Cleaning can also remove sensitive data, such as personally identifiable information (PII), that could create legal or reputational risks if fed into the AI engine.

Choosing a GenAI provider with a robust search solution is a crucial part of quieting data noise. The most effective search systems use hybrid search, which combines an embedding model for semantic, context-driven retrieval combined with lexical matching for precise, keyword-based queries.

Strong embedding models are compatible with a variety of data formats and structures, such as structured (tabular), semi-structured (JSON, XML etc.) and unstructured (text). The best embedding models are trained not only on semantic information, they also reserve numbers in each vector for content quality, structure, and size. This sharpens AI model responses by focusing on higher quality and more relevant data. For additional precision, semantic rerankers can be deployed to order retrieved data based on their relevance to a search query.

What about when documentation contains conflicting information? It may not be possible, or desirable, to expunge all conflicting data. One option is to allow the large language model (LLM) to sift through the data and generate an answer. Another approach would be to tailor the LLM to inform users when a conflict arises and provide different potential answers. Models that include citations enable users to immediately verify the documentation being used to generate an answer. Human experts can guide the LLM to value one type of data source over another, or data with more recent time stamps or written in a certain tone (more neutral, for example) to improve the accuracy of results.

Managing data held in different formats and locations

Data noise also gets amplified by the wide range of structures used to store documentation, from SQL tables to Excel spreadsheets, to CSV files, to XML files. In addition, a lot of data tends to be stored in unstructured formats like emails, PDF documents, and PowerPoint files. If an AI system isn’t able to cope with a wide range of data formats and locations, its effectiveness and capacity to scale will suffer.

Careful consideration needs to be taken for how to access and index various data formats. Once you know what data you need to access, it needs to be made available to the AI model. This requires setting up data connectors that connect an AI model to the relevant document sources and structure type.

The most common enterprise datastores, such as Google Drive, come with pre-built APIs that businesses can use. For less common sources of data, enterprises may need to build their own custom APIs or use an AI provider that includes data connectors as part of their service. Cohere, for example, has built about 100 quickstart APIs that can be used to connect the most popular datastores to our...