Evolution Of Llms Omni World Models

Evolution of LLMs: Omni-World Models

← Back to Blog

Sep 17, 2026

Evolution of LLMs: Omni-World Models

Evolution of LLMs: Omni-World Models

Large Language Models (LLMs) have become a mature technology that can serve multiple products. However, LLMs were never the endgame; they are just the intermediate step. Physical-world intelligence demands an architecture that unifies all possible inputs and outputs, where all the sensory signals, language, images, audio, or videos, can be processed by exactly the same architecture. Even though the recent AI progress mainly meant training bigger models on more textual data, that direction isn’t sufficient if we consider the physical world, where the rules of motion and geometry need to be understood. The gap between what LLMs offer and what’s needed to successfully operate in the real world begs the question of what the next model could look like. Therefore, we’re building “omni world models” that can simulate, and reason about the real world.

What “omni” actually means Our omni model is a single architecture that handles many kinds of input, e.g., language, images or videos, and also produces many kinds of output, using one unified system rather than an orchestration of different agents. To operate with arbitrary known outputs, it has to handle both discrete (like the words in a sentence) and continuous signals (like smooth waveforms of a spoken sentence, or the pixel values in the image).[1] Therefore, our long-standing objective extends beyond generating images, video, and audio within a single architecture. Since an image can be interpreted as a single-frame video, and audio can be represented by a spectrogram (an image), a single diffusion-based model should be capable of handling these modalities. Optimally modelling both types of the signal is the difficult part, and most existing architectures model discrete signals autoregressively or use diffusion-like approach to model continuous ones. While early hybrid models have shown that both approaches can co-exist, building a unified world model that is production-ready is the next frontier of AI. Getting such a model is what makes “omni” special. How “omni” differs from the multimodal models You might be asking then: don’t existing VLMs already handle images and language? Yes, but the key lies in multimodal outputs. VLMs typically transform input signals from various modalities such as images or language into a single output modality (usually language). Reasoning about visual data is what sets them apart from LLMs and enables applications such as OCR, healthcare or quality control. While VLMs can also be applied to robotics, their intuitive understanding of the real-world physics remains a potential constraint . Omni models generalises VLMs to work with potentially arbitrary output modalities, though our initial version specifically focuses on images, videos, language, and actions. In other words, while VLMs usually operate on language, image -> language, our omni model operates on language, image, video, actions -> language, image, video, actions. VLM

LANGUAGE

IMAGE

→

LANGUAGE

IMAGE

VIDEO

ACTIONS

OMNI

LANGUAGE

IMAGE

VIDEO

ACTIONS

→

LANGUAGE

IMAGE

VIDEO

ACTIONS

Same doors in. It is the way out that changes.

VLM

LANGUAGE

IMAGE

↓

LANGUAGE

IMAGE

VIDEO

ACTIONS

OMNI

LANGUAGE

IMAGE

VIDEO

ACTIONS

↓

LANGUAGE

IMAGE

VIDEO

ACTIONS

Same doors in. It is the way out that changes.

Omni implies many new possibilities: Unification of generation and understanding. Most existing approaches rely on agentic methods, where images are understood by one component, but they are generated by another component. Omni folds both into a single process, therefore avoiding costly “swapping” between the models, or cumbersome communication between both systems that have never been trained to live together. Shared representations. Similar to VLMs, image, video or language tokens are all projected into the interconnected representation where the model can reason across all of them at once. With a rough analogy from biology: our brains do not separate “eye language” and “ear language.” Light hitting retina, and wave hitting cochlea both get converted into the same shared representation: electrochemical signals. Thus, perception is unified at the input side. What’s interesting is that generation may also work in reverse. Omni acts as a computational realization of functional integration, using a shared representation to connect diverse inputs and outputs; that is, one shared room, with many doors into that room, and many doors out of that room. Handling continuous and discrete signals. Discrete signals represent language, or logic; both are at the core of communication and reasoning. On the other hand, continuous signals are closer to the physical world: audio, images, videos, or arm joint configurations. The ability to model both within a single model is creating a universal canvas where the model could potentially operate on all domains in science or engineering. “Omni” differs from agentic AI There is a fair objection to the omni approach. Can’t we achieve exactly the same by chaining different agents (either LLMs or VLMs or text-to-video generations)? The current trend of chaining different agents to operate on various modalities is a temporary patch rather than a proper solution. The clear drawback of the chaining is the incurred costs on swapping models: it’s slower as each model adds its own latency, it’s more expensive as each model has to be paid separately. Such approach is also arguably suboptimal as each agent was optimised for its own objective rather than the collaboration. A unified omni architecture structurally resolves these inefficiencies by processing multimodal reasoning within a single, shared representation instead of the API calls. Note that both concepts are also not mutually exclusive as agents can also run omni-world models internally. CHAINED AGENTS

01

understand

↓

HAND-OFF

02

generate

↓

HAND-OFF

03

act

HAND-OFFS

2

LATENCY

cumulative (context switching + 3 models)

BILLED

3x full price (3 separate model invocations)

OBJECTIVES

3, none shared

ONE BACKBONE

understand · generate · act

ONE SHARED REPRESENTATION

NO HAND-OFF

HAND-OFFS

0

LATENCY

zero hand-off latency

BILLED

amortized (pay once for shared memory)

OBJECTIVES

1, joint

CHAINED AGENTS

01

understand

↓

HAND-OFF

02

generate

↓

HAND-OFF

03

act

HAND-OFFS

2...