NousResearch/pokemon-agent

NousResearch/pokemon-agent

Description: AI-powered Pokemon gameplay agent with headless emulation, REST API, and live dashboard. Works with any LLM.

Language: Python

License: MIT

Stars: 138

Forks: 17

Open issues: 2

Created: 2026-03-05T09:30:41Z

Pushed: 2026-06-02T15:21:44Z

Default branch: main

Fork: no

Archived: no

README:

🎮 pokemon-agent

AI-powered Pokémon gameplay agent with headless emulation, REST API, and live dashboard.

Let any AI agent — Hermes Agent, Claude Code, Codex, or your own — play Pokémon games autonomously via a clean HTTP API. Runs headlessly on any server or terminal. No display, no GUI, no emulator window needed.

┌──────────────────────┐
│ Your AI Agent │ Any LLM-powered agent
│ (Hermes, Claude, │ makes the decisions
│ Codex, custom) │
└─────────┬────────────┘
│ HTTP API
┌─────────▼────────────┐
│ pokemon-agent │ This package:
│ ┌────────────────┐ │ - Headless emulator
│ │ Game Server │ │ - Memory reader
│ │ (FastAPI) │ │ - Game state parser
│ ├────────────────┤ │ - REST + WebSocket API
│ │ Emulator │ │ - Optional dashboard
│ │ (PyBoy/PyGBA) │ │
│ └────────────────┘ │
└──────────────────────┘

Features

• 🔌 Headless emulation — No display server, X11, or GUI needed. Pure in-process emulation.
• 🌐 REST API — GET /state, POST /action, GET /screenshot — control the game over HTTP.
• 🗺️ Ground-truth navigation — RAM-derived collision map (GET /map/ascii) and a labelled A1..J9 grid overlay (GET /screenshot/grid) so an agent navigates from real walkability data instead of guessing from pixels.
• 📡 WebSocket — Real-time event streaming for live monitoring.
• 🧠 Structured game state — RAM is parsed into clean JSON: party, bag, badges, map, battle, dialog, collision grid.
• 🎨 Live "Field Log" dashboard — Editorial broadcast UI: the agent's reasoning stream, live grid map, objectives, telemetry (stuck-meter, blackout counter), and a milestone timeline.
• 🎮 Multi-game — Supports Game Boy (Pokémon Red/Blue) via PyBoy, GBA (FireRed) via PyGBA.
• 🤖 Agent-agnostic — Works with any AI agent, RL framework, or custom script.

Quick Start

Installation

# Core (emulator + API server)
pip install pokemon-agent pyboy

# With dashboard (optional web GUI)
pip install pokemon-agent[dashboard] pyboy

> Note: You must provide your own ROM file. This package does not include any game ROMs.

Start the Server

pokemon-agent serve --rom path/to/pokemon_red.gb

╔══════════════════════════════════════╗
║ 🎮 Pokémon Agent Server ║
╚══════════════════════════════════════╝
Game: Pokemon Red
ROM: pokemon_red.gb
API: http://localhost:8765
Dashboard: http://localhost:8765/dashboard
WebSocket: ws://localhost:8765/ws

Game sessions — new game / load game

A *game session* is one named playthrough that binds three things together: the Hermes brain (its session id, so memory carries across turns), the emulator save-states (game progress), and objectives + milestones + stats — all persisted under /games//.

From the dashboard GAME panel: + NEW starts a fresh game (resets the emulator to a clean boot + new manifest + new Hermes brain), LOAD lists past sessions and restores one — its latest save-state *and* the same Hermes session it was played with. Or via the API:

curl -X POST localhost:8765/games/new -d '{"name":"Nuzlocke run"}' # new game
curl localhost:8765/games # list sessions
curl -X POST localhost:8765/games//load # load one
curl localhost:8765/games/current # active session

Saves, objectives, milestones, and stats are automatically scoped to the active session, and the autopilot binds to it — so loading a game resumes exactly where that run (and its Hermes memory) left off.

Autopilot — Hermes Agent plays itself

The server is a passive API — it holds the emulator but does not play. To make the game play autonomously, run the bundled driver in a second process:

# 1. start the server (terminal A)
pokemon-agent serve --rom path/to/pokemon_red.gb

# 2. start the driver (terminal B) — requires the `hermes` CLI on PATH
pokemon-agent play --port 8765

The brain is a real Hermes Agent session, not a bare LLM. Each turn the driver invokes hermes chat --resume --yolo -s pokemon-player --image -q "", so Hermes plays with its full stack — the pokemon-player skill, vision, memory, and the terminal tool — and keeps context across the whole run via one persistent session. Hermes itself curls the server to POST /action, /event (narration), and /objectives.

The driver idles until you press START on the dashboard (it polls /control); START / PAUSE / STOP drive it live. Hermes uses its own configured provider/model; override per-run with POKEMON_HERMES_MODEL / POKEMON_HERMES_PROVIDER. Each turn is a full agent loop (seconds-to-minutes), not a single API call — this is genuine agentic play, not a tight poll.

Play from Any Agent

# Get game state
curl http://localhost:8765/state | python -m json.tool

# Take a screenshot
curl http://localhost:8765/screenshot -o screen.png

# Send actions
curl -X POST http://localhost:8765/action \
-H "Content-Type: application/json" \
-d '{"actions": ["walk_up", "walk_up", "press_a"]}'

# Save/load state
curl -X POST http://localhost:8765/save -d '{"name": "before_brock"}'
curl -X POST http://localhost:8765/load -d '{"name": "before_brock"}'

Game State (JSON)

{
"player": {
"name": "ASH",
"money": 3000,
"badges": 1,
"badges_list": ["Boulder"],
"position": {"map_id": 1, "map_name": "PALLET TOWN", "x": 7, "y": 5},
"facing": "down",
"play_time": {"hours": 1, "minutes": 23, "seconds": 45}
},
"party": [
{
"nickname": "SQUIRTLE",
"species": "Squirtle",
"level": 12,
"hp": 33,
"max_hp": 33,
"moves": ["Tackle", "Tail Whip", "Bubble"],
"status": null,
"types": ["Water"]
}
],
"bag": [{"item": "Potion", "quantity": 3}],
"battle": null,
"dialog": {"active": false, "text": null},
"flags": {"has_pokedex": true, "badges_earned": ["Boulder"]},
"metadata": {"game": "Pokemon Red", "frame_count": 12345}
}

Actions Reference

| Action | Description | |--------|-------------| | press_a | Press A button (10 frames press + 20 wait) | | press_b | Press B button | | press_start | Press Start button | | press_select | Press Select button | | walk_up | Walk one tile up (16 frames + 8 wait) | |…