Gemini 4 Argon for Robotics

Gemini 4 Argon for Robotics

• Google Developers Blog

Gemini 4 Argon for Robotics

OCT. 13, 2026

Kanishka Rao

Principal Software Engineer

Share

Facebook

Twitter

LinkedIn

Mail

Gemini 4 Argon marks a step change in physical understanding, code generation, and visual-semantic reasoning—unlocking new workflows across the robotics stack and accelerating progress toward physical AGI. You can see this leap in capability on the Embodied Reasoning (ERQA) benchmark , which tests a model's ability to understand and solve physical, spatial problems in 3D environments. Using pure reasoning, Argon scores 87%, outperforming every other frontier model.

As a general-purpose physical reasoning model, Argon can be applied across the entire robotics lifecycle. In this post, we share representative prompts for Argon across four key areas:

Robot control → Mobile manipulation with MuJoCo practice, zero-shot robot control, and closed-loop steering of video world models. Real2Sim → Reconstructing physically calibrated simulation twins from robot camera feeds, egocentric human video, or hand-drawn paper sketches. Physical understanding → Agentic multimodal hindsight labeling including detailed understanding of dexterity, dense value functions and progress estimation. Research and self-improvement → Action improvement via simulation and iterative self-improvement on hardware.

Unlocking robot control with Argon 1. Spot mobile manipulation: Putting in the trash bin with MuJoCo practice Argon controls a Boston Dynamics Spot® quadruped (12-DoF legs + 6-DoF arm) to learn a new skill: stepping on a foot-pedal trash bin to throw away soda cans. Argon creates an articulated MuJoCo digital twin of a foot-pedal trash bin to practice its moves before stepping on the physical pedal and dropping soda cans into the open bin.

Sorry, your browser doesn't support playback for this video

Spot Mobile Manipulation — Real-to-Sim-to-Real foot-pedal trash bin disposal: Spot picking up the soda can, checking its wrist camera, practicing 3-leg tripod balance in MuJoCo, stepping on the foot pedal, verifying hand centering in the open bin, and dropping the can inside.

Prompt:

There's a Diet Coke can behind you and another on the black table on your left. Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin. Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.

Plain text

Copied

2. Zero-shot control Argon controls a simple robotic arm to place a fallen mug upright and put the crumpled napkin inside. Argon reasons with a visual coordinate harness and action primitives to perform this task over several turns.

Sorry, your browser doesn't support playback for this video

Left: 3rd-person real robot view; Right: calibrated wrist camera with projected 2 cm world coordinates and jaw-closing reticle.

Prompt:

[Attached: wrist_cam.png, obs.json]

Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.

Execution Discipline

• Simply output 1 sentence observation followed immediately by your curl command (with # Plan: on Turn 1 and # Thought: on every turn). Everything needed is in obs.json and the calibrated reticle/grid drawn on wrist_cam.png.

Visual Coordinate Reference (Annotated on wrist_cam.png)

• +X (UP in image): Away from robot base | -X (DOWN): Towards robot base
• +Y (LEFT in image): World left | -Y (RIGHT): World right
• +Z: Height above table | Yellow numbers: World (X, Y) in centimetres (2 cm grid at grid_z)
• Cyan + crosshair: Current 3D pinch center
• Cyan [ + ] brackets: Current jaw opening and closing line
• Green line: Current object long-axis orientation (yaw_deg: 0° = vertical along +X, 90° = horizontal along +Y)
• pitch_deg = 90°: Straight down (-Z) top-down grasp/place; pitch_deg grasp -> place -> done`
• Pre-Grasp Hover Verify (4–7 calls): look_at -> goto(z=0.04) (verify cyan [ + ] brackets straddle target) -> grasp -> place_until_contact -> done
• Fine Visual Servoing (8–20 calls): look_at -> grasp -> pre-pose goto -> closed-loop move_by / move_until_contact -> done

3. Send motion primitives (POST /plan) and verify visually after every turn in the updated wrist_cam.png. If status == "grasp_failed", call robot.look_at(...) to re-observe and recover closed-loop.

Code-as-Policies Primitives (POST http://127.0.0.1:8081/plan)

robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015) x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015) robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90) robot.place(x=0.24, y=0.06, z=0.075) robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002) robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10) robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04) robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002) robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()

Plain text

Copied

3. Omni control Argon steers robots in the Gemini Omni World Model to rearrange five scattered Scrabble tiles ( G, R, A, N, O ) into a clean row spelling A - R - G - O - N . Instead of specifying one static prompt, Argon steers in a closed visual loop. It writes timestamped action prompts, inspecting intermediate rollout keyframes, and refining its prompts.

Sorry, your browser doesn't support playback for this video

Iterative Video World Model Steering Dexterous Hands: 33-Step Steered Video Rollout Spelling A-R-G-O-N.

Prompt:

Goal: {task_goal}

You are steering a video world model (Gemini Omni) in an iterative visual feedback loop. Inspect the attached frames from the latest rollout and write the next timestamped action prompt for the video world model:

• Break complex motions into one simple, single-direction action at a time.
• Keep inactive hands retracted and avoid blocking the camera's view of objects in the workspace.
• Explicitly state which areas of the workspace become empty after an object moves.
• Use [MM:SS.mmm] timestamps for each action beat and wrap your output in tags.

Example format (sample timestamps only, not the full prompt):

[00:00.000]...