amazon-science/MemInsight

amazon-science/MemInsight

Language: Python

License: NOASSERTION

Stars: 8

Forks: 1

Open issues: 3

Created: 2025-09-15T04:55:41Z

Pushed: 2025-09-15T05:17:13Z

Default branch: main

Fork: no

Archived: no

README:

MemInsight: Autonomous Memory Augmentation for LLM Agents

Authors: Rana Salama, Jason Cai, Michelle Yuan, Anna Currey, Monica Sunkara, Yi Zhang, Yassine Benajiba

MemInsight is a structured memory augmentation framework designed to enhance the long-term reasoning and adaptability of large language model (LLM) agents. It introduces autonomous memory annotation and retrieval methods that help agents organize and access relevant historical context during inference.

---

🔍 Overview

As LLM agents scale, managing accumulated memory across diverse interactions becomes a major challenge. MemInsight addresses this by:

• Autonomously generating structured memory augmentations
• Prioritizing semantically rich attributes for retrieval
• Improving memory relevance with attribute-based and embedding-based methods
• Boosting response quality in recommendation, QA, and summarization tasks

---

🧩 Key Features

• Attribute Mining: Extracts entity- and conversation-centric attributes from dialogues
• Memory Annotation: Supports both turn-level and session-level augmentation
• Retrieval Methods:
• Attribute-Based Filtering
• Embedding-Based Similarity (FAISS)
• Task Support:
• Conversational Recommendation
• Question Answering
• Event Summarization

---

📈 Benchmark Results

MemInsight outperforms traditional memory retrieval methods:

| Task | Metric | Improvement | |-------------------------|------------------|-------------| | QA (LoCoMo) | Recall@5 | +34% over DPR | | Conversational Reco. | Persuasiveness | +14% | | Event Summarization | G-Eval (Relevance) | Comparable to baseline with less memory |

---

📂 Datasets Used

• LLM-REDIAL: Movie recommendation dataset
• LoCoMo: Multi-turn long-context QA and summarization

---

🛠️ Setup

git clone https://github.com/amazon-science/MemInsight

cd meminsight

pip install -r requirements.txt

Run attribute mining and augmentation

python main.py --dataset llm-redial --model claude-3-sonnet

Evaluate

python main.py --task recomm --dataset "dataset_path" --anotations "annotations path"

---

Models Used

• Claude-3 Sonnet / Haiku (Augmentation & Generation)
• LLaMA 3 (Alternative Backbone)
• Mistral (Low-resource variant)
• Titan Text Embedding (FAISS indexing)

---

📊 Evaluation Metrics

• QA: F1, Recall@K
• Movie Recommendation: Recall@K, NDCG, Genre Match, Persuasiveness
• Event Summarization: G-Eval (Relevance, Coherence, Consistency)

---

Paper

This repository implements experiments and methods from the paper: “MemInsight: Autonomous Memory Augmentation for LLM Agents” ACL 2025 Submission (Under Review) 📌 Source code and data samples will be released upon acceptance.

---

Citation

If you use this code or refer to MemInsight in your work, please cite:

@misc{salama2025meminsightautonomousmemoryaugmentation,
title={MemInsight: Autonomous Memory Augmentation for LLM Agents},
author={Rana Salama and Jason Cai and Michelle Yuan and Anna Currey and Monica Sunkara and Yi Zhang and Yassine Benajiba},
year={2025},
eprint={2503.21760},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2503.21760},
}

---

Security

See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.

License

This library is licensed under the CC-BY-NC 4.0. See LICENSE file.