How Hamlet uses Databricks Lakebase Postgres to test changes against production data

How Hamlet uses Databricks Lakebase Postgres to test changes against production data | Databricks Blog Skip to main content

Summary

Hamlet gives every pull request its own Lakebase Postgres branch of production data, so engineers test changes against the data that caused a bug without touching production.

AI coding agents run the pipelines and check their work on each branch before a person approves the merge.

Hamlet is exploring Lakebase Search for vector and text search in Postgres, branching for object storage and a home for longer agentic jobs.

Hamlet, an AI-powered platform that makes local government decision-making accessible and searchable, uses Databricks Lakebase Postgres to give every pull request its own branch of production data. Engineers and AI coding agents test changes against real data without touching production, and a person still approves every merge. I spoke with CEO Sunil Rajaraman, Lead Developer Erich Douglass and Senior AI Engineer Jeff Hui about how they work. What is Hamlet? Hamlet is an AI-powered technology platform that makes local government decision-making accessible and searchable. By indexing and transcribing thousands of city council, planning commission, and zoning board meetings across the U.S., Hamlet allows real estate developers, journalists, researchers, and citizens to instantly search meeting transcripts, track local policy developments, and set up real-time alerts for specific topics, addresses, or businesses. Why is local government data so hard to standardize? Sara Steffen: Hamlet indexes meetings from more than 7,200 governing bodies across every state. What does that data landscape look like, and why is it so hard to make sense of? Sunil Rajaraman: There are 19,000 city governments, 3,000 county governments and roughly 80,000 special districts in the United States. Pick any two of their websites and they won’t look alike. That’s the original sin of local government. A handful of CMS providers decided every site should be different, and the result is an internet of government that’s completely balkanized. Hamlet pulls all of it together and standardizes it. How is a civic intelligence platform different from a transcription tool? Sara: Hamlet describes itself as a civic intelligence platform. How is that different from a transcription tool, and why does the distinction matter? Sunil: A lot of govtech companies are an arbitrage of municipal data. They write brittle scrapers for 1998-era websites and put a modern coat of paint on top. Scraping and storing the data doesn’t add much on its own. You have to tell a story, and it has to be based on fact. That means knowing who’s speaking and how they feel about an issue. If you’re a developer, you want to know whether a council member will approve your housing project, why or why not, and what history the city has that might make it hesitate. Look at permitting. A lot of venture money has gone into permitting data, and permitting timelines have still gotten worse over the past decade. Many companies have built monuments to the symptoms. We want to get to the underlying problem, which is understanding people. The question we’re trying to answer is how cities fundamentally work. What makes messy public data so hard to work with? Sara Steffen: What makes local government data so difficult to work with? Sunil Rajaraman: Pick any two of their websites and they won’t look alike. Hamlet pulls it all together and standardizes it. Erich Douglass: You can’t just integrate with one provider. Seven or eight of them cover about 75% of the market. After that it’s a massive long tail of custom setups. Sometimes a clerk uploads a PDF of the agenda. Sometimes they scan a paper copy and upload that. You get some really weird artifacts. What do customers need from meeting data? Sara: What do customers need from that data? Sunil: Our customers are private companies like real estate developers, data center developers and retailers. Nearly all of them are trying to get a project approved faster or find a new opportunity. Without Hamlet, someone sits through hours of meetings, waits for their project to come up and watches the officials’ body language. We identify who spoke, take you to the timestamp and show you the receipts. Erich: They also want precedent. Has a project like theirs been approved or rejected before? Why? What changes did the council ask for? If we can show what’s really going on, that’s a lot of value. How does Hamlet keep AI answers tied to the source? Sara: How do you help customers trust the insights they find? Jeff Hui: We use it in almost every part of the process. Early on, Hamlet did this work by hand, and that doesn’t scale to thousands of places. For a lot of these governments, the video is the primary source of truth. Getting it into a structured form we can use is the most important part. The average meeting runs about three and a half hours. Nobody wants to watch all of it. AI points people to the parts worth their time, and the video is always there to back it up. We can show exactly what was said, which a lot of AI systems struggle to do. Context makes the difference Sara: How do you identify speakers accurately across thousands of meetings? Jeff: Context makes the biggest difference. For speaker identification, we know which council or committee is meeting and who was on it at the time. That helps with names, which are hard to get right even for a person listening to a recording. You can’t hand a model a video and hope for the best. You have to collect everything around it that helps it make the right call. It’s a forever challenge. Checking AI interpretations Sara: How do you check the AI’s interpretation of what officials say? Sunil: Accurate transcription comes first. For sentiment, we use multiple AI models to confirm a comment is about a given topic, say data centers. Then they look at the surrounding context to decide whether it’s for or against. You also have to define what pro and anti mean before you ask. Take policing. What does being for it mean? We put a lot of care into those questions. Several models now review every sentiment figure before we publish it, and we require high confidence. How does Hamlet use Lakebase Postgres? Lakebase is the managed Postgres database Hamlet uses to give each pull request its own branch of production data. Engineers can test changes against the data that caused a bug without touching production, while AI coding agents run the...