Ai System Requirements

Why Claude Code isn't enough to build AI systems

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

TL;DR

The bottleneck in AI development has shifted. It is no longer about translating natural language into code; it is about translating the human subconscious into a conscious specification.

The end of software development

For decades, we thought coding was the hard part. A product manager would write a PRD, and it would take weeks for developers to turn it into working software. Now, with tools like Claude Code or Cursor, you can go from natural language to Python in minutes.

Sure, everyone is talking about the end of software development, but we haven’t actually removed the bottleneck. We just moved it. And the new one is much harder, especially when the product itself is AI, such as an internal knowledge agent or a customer support chatbot.

The problem today isn’t how to tell the computer to do something. The problem is that humans don’t actually know what they want the computer to do.

The “I know it when I see it” paradox

In 1964, Justice Potter Stewart famously defined hard-core pornography with a single phrase: “I know it when I see it.”

This is exactly how humans experience software requirements. We are terrible at generating abstract rules, but we are excellent at criticizing concrete examples.

If you ask business stakeholders to define requirements for a customer service chatbot, they will write generic platitudes: “Be polite and professional.” But when that agent gives a dry, robotic answer, they immediately scream: “No! Not like that!”

This is the central friction of the Agentic Era: We are trying to define infinite, recursive behavioral systems using finite, linear instructions.

The “Dwight” fallacy: Data ≠ Context

This gap between what we feel and what we can specify is exactly why naïve approaches to AI – like “just give it more data” – consistently fail.

Consider Dwight Schrute, the legendary employee who knows everything about the organization: history, politics, shortcuts and Battlestar Galactica. After a gun shooting accident in the office, Dwight resigns and moves to Staples, and management says: “Let’s build a Dwight-Bot. Just ingest his 20 years of emails into the RAG.”

They build the bot and it fails. Why?

Because Dwight wasn’t valuable for the data he held, but for the context he applied . He knew when to ignore the 2018 policy because “Jeff from Accounting doesn’t care.” Ingesting data captures the text, not the subconscious knowledge of the organization.

The speech act trap: Words ≠ Intent

But even if we solve the “data vs. context” problem, a deeper issue remains: understanding intent, not just information.

Agents often fail even when they answer “correctly”. They understand the words, but miss what the user is actually trying to achieve.

The reason is simple. In language, we don’t just convey information, we perform actions through it (a concept formalized as Speech Act Theory by Austin & Searle ).

Consider the following example:

User: “The printer is jamming again.”

The bad agent (literal focus):

“I understand. Paper jams are often caused by misaligned rollers or debris.”

(Factually true. Pragmatically useless.)

The good agent (intent focus):

“I’ve opened a ticket. Do you want me to route your jobs to the backup printer on the 3rd floor?”

The user didn’t make a statement of fact, they issued a directive. The “bad agent” failed not because of a logic error, but because of a felicity error . It failed to perform the speech act the user actually needed.

The Claude Code illusion: Why “spec-based” AI isn’t enough

These failures aren’t theoretical. They show up directly in how we build AI systems today. Interestingly, the industry is already moving in the right direction.

Tools like Claude Code have introduced features such as AskUserQuestion, allowing the model to actively interview the developer before writing a single line of code. For years, we obsessed over prompt engineering, and now the model prompts you . This acknowledges a deeper truth: building software is not about writing code, it’s about interrogating intent.

However, this creates a dangerous illusion. Claude’s interview tool is designed to resolve technical ambiguity, not conceptual ambiguity. If a business stakeholder says, “ Build a bot based on Dwight’s emails ,” Claude will ask excellent technical questions about vector databases or authentication. It will not ask if Dwight actually followed those written rules, nor will it read the unwritten political subtext of the company. It won’t challenge the premise itself.

To translate humans’ subconscious intents into a working product, you cannot just hand raw requirements to an AI code assistant. You need to actively probe the flawed assumptions, and architect the true requirements before the coding even begins. Otherwise, Claude will indeed help you build more efficiently, but build the wrong thing.

The solution: Active discovery and the AI product architect

To build an AI system that actually works, you need more than a coding assistant. You need an experienced builder acting as an active investigator. This requires a methodology of Active Discovery : translating an organization’s subconscious intent into a concrete, pragmatic PRD. Here’s how that process works:

1. Sources mapping phase: Context & Data

An amateur asks the business stakeholder, “What should the bot do?” An experienced AI builder asks, “What does the bot actually know?”

We map the Context : Is this agent independent? Is it part of a legacy system?

We map the Data : What are the actual sources of truth?

Crucially, the experienced builder actively hunts down the Implicit Gaps . They identify the unwritten rules and context that Dwight kept in his head, and force the organization to explicitly define them before writing a single line of code.

2. The mocking loop: Simulation over specification

It’s hard for humans to accurately simulate complex conversational trees in their heads. If you ask business stakeholders for rules, they will give you useless platitudes. Instead, an experienced AI builder generates Mocks : simulated Input/Output pairs...