Beyond Recognition How Our Models Reason About Video

Beyond Recognition: How Our Models Reason About Video

Reka Responsible AI, Model Risk, Ethics & Governance Framework

← Back to Blog

Jul 30, 2026

Beyond Recognition: How Our Models Reason About Video

Beyond Recognition: How Our Models Reason About Video

Building an omni world model means more than generating a plausible video. The model also has to know not just what a concept is, but where it is (in space and time). That's what spatial and temporal reasoning give a world model: the grounding to act, not just describe. Most models can already tell you that a motorcycle, a person or a bag is on screen. The answers people actually need are the following inferences: where exactly is it, what happened over time, who it is, and does it matter. This post walks through four reasoning types our models perform, with real output from real footage for each, and ends with the part we consider just as important: this reasoning is built to run at the edge, on the devices where the cameras are. Seeing Precisely: Spatial Grounding Useful reasoning starts with knowing exactly where things are, not just that they exist. Our models are trained for precise localisation: given an open-vocabulary instruction such as Detect: red car, man with a hat , Reka Edge returns bounding boxes for every match, and the same grounding extends to pixel-accurate segmentation masks. The masks below are model output on broadcast footage, precise enough to drive compositing effects that would otherwise be rotoscoped by hand:

Spotlight effect driven entirely by model-generated masks: subjects in full colour, background dimmed. Image via Wikimedia Commons. Grounding at this precision is what robotics, automotive and visual search applications need, and it is a capability we benchmark directly: Reka Edge scores ~93.01 on RefCOCO-A, ahead of Gemini 3 Pro's 81.46, at a fraction of the size. Qwen-VL-8B-Instruct scores marginally higher on both RefCOCO splits. The gap that matters here is size, not the top spot. Following Time: Temporal Reasoning A frame tells you what is visible. Only a sequence tells you what is happening. Our models reason over time: they can find extended activities such as assembling a desk in hours of footage, answer questions anchored to timestamps such as what happened between minute two and three, and turn raw video into per-subject timelines of exactly when each subject is on screen:

A per-subject appearance timeline generated from tracking output: clicking a band seeks the video to that appearance. Image via Wikimedia Commons. Temporal reasoning is where small models usually break down. Reka Edge scores ~72.31 on MLVU and ~72.80 on MMVU, roughly double Cosmos Reason 2 and clearly ahead of Qwen-VL-8B-Instruct, while approaching Gemini 3 Pro, the leading long-video understanding benchmarks. Knowing Who: Identity as Inference The hardest question in video is who, asked continuously across camera cuts, occlusions and near-identical appearances. Identity is not a detection. It is an inference assembled from weak signals: appearance embeddings, evidence accumulated across every clip a subject appears in, opportunistic reads of numbers or text aggregated by voting, and context constraints such as a known roster and the fact that two subjects in one frame cannot share an identity. We stress-tested this on professional motorsport broadcast footage, close to a worst case: twenty subjects at high speed, where teammates ride visually identical machines and vision-language models prompted with the full roster cannot separate them. On a frozen, pre-registered benchmark of footage the model had never seen, the composed pipeline names competitors from pixels alone with 90.9% top-1 accuracy (92.4% macro, Wilson 95% interval [76.4%, 96.9%]), with leave-one-clip-out evaluation so the queried clip is never in the competitor's own reference gallery. The gains over the 39.6% baseline came from evaluation hygiene and matcher changes, not larger models.

Held-out race start: every bike detected, tracked and matched to a rider, with the overlay rendered automatically. Image via Wikimedia Commons.

Cross-clip clusters (excerpt): the same bike grouped across cuts by whole-bike embedding and livery hue. Image via Wikimedia Commons. The honest caveats: these results come from a single event weekend, tracking degrades on distant aerial shots, and identical-livery teammates remain the hardest case. The same machinery scales from naming twenty competitors to managing crowds of thousands. Crowd density in a train station sounds like a counting problem, but useful crowd intelligence is an identity problem: a count of 23 people in one frame cannot say whether a platform is clearing or filling. Continuous tracking can, because when every passenger keeps a stable identity across frames and cameras, the system knows how long each person has been waiting, whether the platform cleared between trains, and where the crowd is flowing. That is the difference between a number on a dashboard and a signal an operator can act on, such as deploying staff before crowding becomes dangerous.

Judging What Matters: Anomaly and Intent The final reasoning type is judgment: most of what a camera sees is routine, and the value is in recognising the moment that is not. This requires composing the other three. A door opening is normal. The same door opening at 3 AM, followed by a person concealing their face, is an anomaly, and telling those apart requires spatial, temporal and identity reasoning working together. On SmartHome-Bench, a video anomaly benchmark developed by Wyze spanning seven categories from security events to wildlife, our models set the leading score where competitors hallucinate events, miss temporal context or fail to explain why an event is anomalous. Example:

A security clip from SmartHome-Bench: a strange man approaches a car and inspects its components. Our Reka Vision is able to judge this scenario as anomalous. The same judgment applies beyond the home. In rail transit, the anomalies that matter most are threats to safety: a knife or other weapon appearing in a crowd, a fight breaking out on a platform, a passenger slipping and falling while boarding, a person climbing over the platform doors into the track area. Each of these is rare, unfolds in seconds, and is expensive to miss. Each is also a composition of the other reasoning types: spotting a weapon is spatial grounding, telling a fight from a greeting takes...