google-deepmind/life-bench

google-deepmind/life-bench

Description: Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

Language: Python

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-09-04T20:51:52Z

Pushed: 2026-10-08T19:45:14Z

Default branch: main

Fork: no

Archived: no

README: Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

Xia Hu, Honglei Zhuang, Brian Potetz, Alireza Fathi, Bo Hu, Babak Samari, Howard Zhou

Google DeepMind

Overview

Life-Bench is a fully synthetic, human-verified multimodal benchmark of 11,811 question–answer pairs across 10 tasks, organized by required evidence scope: Concept Identification, Event Understanding, and Aggregated Reasoning. Underlying these tasks, the personal context comprises social networks and multimodal lifelogs organized into 10 isolated virtual accounts (Vaccounts) to simulate individual users' archives; every question and image has been verified through full-coverage human annotation. All data in Life-Bench (images, social networks, and event histories) is synthetically generated, enabling public release without exposing real users' data.

Please refer to the paper for the benchmark design, construction pipeline, and quality control. This repository contains the evaluation code; the dataset is hosted on the Hugging Face Hub at google/life-bench.

Tasks

| Category | Task | Task Key | Input Format | Answer Format | # Questions | | :--- | :--- | :--- | :---: | :---: | ---: | | Concept Identification | Text Concept QA | text_concept_qa | Text | Multiple-Choice | 163 | | | Visual Concept Recognition | visual_concept_recognition | Text + Image | Binary | 1,613 | | | Concept VQA | concept_vqa | Text + Image | Multiple-Choice | 885 | | Event Understanding | Scene and Activity | scene_and_activity | Text | Open Generation | 2,265 | | | Direct Person-Centric | direct_person_centric | Text | Open Generation | 2,245 | | | Relational Person-Centric | relational_person_centric | Text | Open Generation | 1,812 | | | Fine-Grained Scene | fine_grained_scene | Text | Open Generation | 1,981 | | Aggregated Reasoning | Preference and Persona | preference_and_persona | Text | Open Gen (Option-guided) | 295 | | | Frequency and Counting | frequency_and_counting | Text | Open Generation | 265 | | | Relational Temporal Reasoning | relational_temporal_reasoning | Text | Open Generation | 287 | | Total | | | | | 11,811 |

Evaluation Results

Our primary metric is accuracy: multiple-choice and binary questions use exact match, while Gemini 3.6 Flash judges open-ended responses against the ground-truth answer using LLM-as-a-Judge. We additionally report recall to assess retrieval quality.

Accuracy. Retrieval-based methods are compared at context size $k = 5$, all using Gemma-3 12B as backbone; the best and second-best scores among them are in bold and underlined respectively. Reference settings contextualize retrieval performance and are generated with Gemini-3.5-Flash except Random: Close-book answers without personal context; Oracle receives only the gold supporting evidence (Oracle-Gemma: with the Gemma backbone); Full-context places an entire Vaccount history in the prompt.

| Method | Text Concept QA | Visual Concept Recognition | Concept VQA | Scene & Activity | Direct Person-Centric | Relational Person-Centric | Fine-Grained Scene | Preference & Persona | Relational Temporal | Frequency & Counting | Overall | | :--- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | | Random | 0.2500 | 0.5000 | 0.2500 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.2003 | 0.0000 | 0.0000 | 0.1200 | | Close-book | 0.2454 | 0.5111 | 0.2696 | 0.0848 | 0.0539 | 0.1341 | 0.1560 | 0.4373 | 0.0662 | 0.0755 | 0.2034 | | Full-context | 0.9730 | 0.9722 | 0.9508 | 0.9143 | 0.9095 | 0.9141 | 0.9014 | 0.8856 | 0.8523 | 0.3823 | 0.8655 | | Oracle | 0.9816 | 0.9826 | 0.9570 | 0.9324 | 0.9292 | 0.9269 | 0.9100 | – | – | – | 0.9457\* | | Oracle-Gemma | 0.8527 | 0.8698 | 0.6170 | 0.8093 | 0.7007 | 0.7130 | 0.7410 | – | – | – | 0.7576\* | | BM25 | 0.8650 | 0.8320 | 0.6249 | 0.6698 | 0.3648 | 0.4923 | 0.4962 | 0.5593 | 0.1080 | 0.1472 | 0.5159 | | RAG-Cap | 0.8405 | 0.7650 | 0.5236 | 0.4843 | 0.2895 | 0.3786 | 0.5671 | 0.7051 | 0.1428 | 0.1623 | 0.4859 | | RAP-Gemma | 0.7651 | 0.7498 | 0.4744 | 0.3043 | 0.1688 | 0.2467 | 0.5172 | 0.7119 | 0.1533 | 0.0755 | 0.4167 | | R2P-Gemma | 0.7055 | 0.6943 | 0.3559 | 0.3244 | 0.2664 | 0.3488 | 0.4338 | 0.5831 | 0.1715 | 0.1094 | 0.3993 | | HippoRAG2 | 0.8098 | 0.8494 | 0.6542 | 0.3978 | 0.2094 | 0.3201 | 0.4356 | 0.5559 | 0.1672 | 0.1623 | 0.4562 | | LifeGraph ($d=2$) | 0.7587 | 0.8332 | 0.5318 | 0.7242 | 0.5092 | 0.5193 | 0.5157 | 0.6746 | 0.3240 | 0.1785 | 0.5569 | | LifeGraph ($d=3$) | 0.7914 | 0.8278 | 0.5260 | 0.7228 | 0.4984 | 0.5348 | 0.5253 | 0.6524 | 0.3275 | 0.1976 | 0.5604 |

\* Oracle overall averages over seven concept and event tasks since aggregate does not have subset golden reference.

Event-level Recall@3/5, where retrieving any image from a gold event counts as a hit.

| Method | Concept R@3 | Concept R@5 | Event R@3 | Event R@5 | Overall R@3 | Overall R@5 | | :--- | :-: | :-: | :-: | :-: | :-: | :-: | | BM25 | 0.4626 | 0.5330 | 0.4785 | 0.5371 | 0.4717 | 0.5353 | | RAG-Cap | 0.4307 | 0.4808 | 0.3665 | 0.4580 | 0.3940 | 0.4678 | | R2P | 0.4070 | 0.4238 | 0.3823 | 0.4299 | 0.3929 | 0.4273 | | RAP | 0.4354 | 0.4653 | 0.4065 | 0.4572 | 0.4188 | 0.4607 | | HippoRAG2 | 0.4465 | 0.5128 | 0.3042 | 0.4065 | 0.3652 | 0.4521 | | LifeGraph ($d=2$) | 0.4748 | 0.5009 | 0.5309 | 0.6021 | 0.5068 | 0.5587 | | LifeGraph ($d=3$) | 0.4844 | 0.4991 | 0.5271 | 0.5963 | 0.5088 | 0.5546 |

Dataset

The dataset is hosted at google/life-bench as Parquet tables (questions, history, concepts) and as a raw file tree:

google/life-bench/
├── concepts/{vaccount}/
│ ├── concepts.json # concepts of the Vaccount
│ └── portraits/*.jpg # concept portraits
├── events/
│ ├── metadata.jsonl # one record per historical image (382 events)
│ └── images/*.jpg # 2,479 historical images
├── questions/
│ ├── .jsonl # 10 task files + 3 `_easy.jsonl` files
│ └── images/*.jpg...