Why Do All LLMs Need Structured Output Modes

Why do all LLMs need structured output modes?

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Why Do All Llms Need Structured Output Modes Why do all LLMs need structured output modes?

PUBLISHED 2/20/2024

Tl;dr: All Fireworks language models can now be invoked with either a (a) JSON schema (b) context-free grammar (similar to Llama.cpp’s feature) to guarantee that LLM output strictly follows your desired format without hallucinations. Intro Since ReAct and Gorilla , there have been tentative explorations into tool use for both chatbots and agents. However they were not reliable or scalable and mostly constrained to research, until OpenAI launched ChatGPT plugins last March and then innovated the function calling API in June, becoming the most widely adopted way for LLMs to generate JSON, initially only for function calling, but formalized as an official “JSON Mode” API in November. However, this is only available in OpenAI’s closed platform. Simultaneously, explorations in constraining LLM output proceeded both in academia ( ReLLM , GCD ) and industry ( guidance , guardrails , outlines ), The most flexible and widely used version of LLM constraints eventually came out of Llama.cpp’s grammar-based-sampling feature. However, this was only available locally on certain supported llama.cpp models. We believe that structured output modes for LLMs have gone from research prototypes in 2023 to table-stakes in 2024 . Ensuring that LLMs respond with predictable, parsable output is critical to many use cases. For example, if you’re using an LLM to generate arguments for an API call, you always want the model to respond with a specific API schema. If you’re using an LLM to tag text, you might always want to use the same labels. Developers can spend hours perfecting a system prompt to “only respond in valid JSON'' or “respond in less than 500 words” and still get unintended output. With function calling, you can build even more complex AI engineering abstractions, resulting in leading developers concluding that Pydantic is all you need and Structured Data is the best way to do Chain of Thought . What we’re doing about it Today, we’re launching of two features on all Fireworks language models: JSON mode - Always generate valid JSON according to a provided schema Structured grammar mode * *- Always generate valid output according to arbitrary context-free grammar for maximum flexibility. We take inspiration from the popular grammar feature in Llama.cpp . As we understand, we’re the only** hosted model API provider supporting this feature!

Use both features on the Fireworks generative AI platform to get blazing fast inference and battle-tested reliability! JSON mode JSON mode enables users to define a JSON output schema for the model to follow. In Python, you can get the schema from any Pydantic structure . Similarly, it can be generated from a TypeScript structure . JSON mode allows you to better use any of our models for function calling or other structured output. For example, if you were using the Fireworks Mixtral model to tag characteristics from an email, you would first define the schema you’d want to output. In this example, that might be: '''json class Tag(BaseModel): language: str sentiment: str topic: str ''' We would then specify this output schema when we call the model by specifying it as the response. Note that we also still describe the output format in the prompt. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 chat_completion = client . chat . completions . create ( model = "accounts/fireworks/models/mistral-7b-instruct-4k" , response_format = { "type" : "json_object" , "schema" : Tag . schema_json ( ) } , messages = [ { "role" : "system" , "content" : "Given an email, classify the email's language, sentiment and topic in JSON." , } , { "role" : "user" , "content" : "Looking forward to meeting next Friday! We’ll come prepared with a proposal for project scoping. Can’t wait to discuss with you!" , } , ] , )

The model would then respond according to the specified JSON schema. In this example, we get the output: 1 2 '{\n    "language": "English",\n    "sentiment": "Positive",\n    "topic": "Meeting to discuss project scoping"\n}'

Voila! Every time you call the model, you can rest assured that the output will fit your JSON schema.How does it work? What’s happening behind the scenes is that we’re taking the scheme you provide and forcing the model to decode according to its structure. For each step of the autoregressive decoding, we only allow new tokens to be generated that would be considered valid in the provided schema. See our documentation for more information on how JSON mode works and how to use it! Important : when using JSON or Grammar mode, it's crucial also to instruct the model to produce the desired schema via a system or user message. The description in the prompt may be the same formal schema or it can be in natural language. Grammar mode Outputting JSON is great but not all rules/structure can fit neatly in JSON. Grammar mode lets you describe the desired context-free grammar in an extended BNF form . The possibilities are endless! Anything from output styles to entire programming languages can be represented as grammars. Let’s take a few examples. Use case #1 Let’s say you want to make use of a mistral model to make an initial medical diagnosis. (NOTE: Fireworks does not intend to give medical advice). In this example, you’d want to limit the medical diagnosis to just the name of 1 of 5 conditions. Here is what the grammar would look like: 1 2 3 4 root      : :=  diagnosis

diagnosis : := "arthritis" | "dengue" | "urinary tract infection" | "impetigo" | "cervical spondyl

The root rule is what the entire output has to match. We define the root rule to be just the diagnosis rule. And for diagnosis, we defined it to be one of the five different classes, each separated with a | to signify that it can be any of these words. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 from fireworks . client import Fireworks

client = Fireworks ( api_key = "" , )

diagnosis_grammar = """ root      ::= diagnosis diagnosis ::= "arthritis" | "dengue" | "urinary tract infection" | "impetigo" | "cervical spondylosis" """

chat_completion = client . chat . completions...