Building Domain Agents: The Real Bottleneck Isn't Expert Time — It's Expert Consensus

Where the Real Bottleneck Lies in Domain Agents

When building AI agents (domain agents) for specialized fields like finance and operations, many teams find that preparing evaluation and training data isn't held back by the toolchain. What actually slows things down is getting the "right answers" from people who truly understand the business.

This observation comes from a Reddit user's conversations with practitioners building agents for finance and operations teams. Their conclusion is telling: the constraint on agent deployment is often not technical capability, but how domain knowledge is captured and crystallized. When we talk about AI applications in vertical domains, this problem is consistently underestimated.

Expert Time vs. Expert Consensus

A distinction emerged from these discussions that's worth sitting with. One person put it bluntly: getting time with a Subject Matter Expert (SME) isn't actually that hard — the slow part is getting them to "agree on what the right answer even is."

This reframes the problem from a resource issue to a cognitive one. You can easily pull three senior analysts into a meeting room, but when faced with the same edge case, they might offer three different ways of handling it — and each one holds up within its own experiential framework. For training data and evaluation benchmarks (evals), this kind of disagreement is fatal: machine learning requires definitive labels, while real-world business is full of context-dependent judgment calls.

Why Experts Disagree

Domain knowledge is often tacit. Experienced practitioners develop intuitions over years of practice, but those intuitions are never explicitly encoded as rules. When asked to make their judgments explicit, different experts reveal their varying interpretations of rule boundaries, edge cases, and risk tolerance. These differences normally hide in the unspoken "everyone just knows" layer of organizational culture — they only surface when you try to label data for an agent.

Cognitive science calls this kind of knowledge tacit knowledge, a concept introduced by Michael Polanyi in the 1960s. Unlike explicit knowledge that can be written into a manual, tacit knowledge is embedded in a practitioner's experience, intuition, and embodied understanding — it's inherently difficult to verbalize. In the AI training data space, this is known as the knowledge elicitation bottleneck: domain experts genuinely possess the right answers, but converting those answers into a format machines can learn from is itself an enormously difficult task.

Traditional knowledge engineering already ran into this wall during the expert systems era of the 1980s and 90s, developing methods like structured interviews and think-aloud protocols to extract tacit knowledge — none of which fundamentally solved the problem. The arrival of large language models hasn't eliminated this bottleneck; if anything, it has amplified it by raising the quality bar for labeled data.

The eval (evaluation benchmark) mentioned here plays a role in LLM application development similar to unit tests in software engineering — it's a test set with ground-truth answers used to measure whether an agent's performance on specific tasks meets the standard. Unlike general-purpose benchmarks (such as MMLU or HumanEval), domain agents require custom evals built from real business scenarios, and their quality directly determines whether the entire development iteration is pointed in the right direction.

If the ground-truth answers in the eval are themselves contested or incorrect, a team may pour significant compute and labeling resources into optimizing for the wrong objective — ultimately producing a model that scores well on the benchmark but fails regularly in actual business use. This is a textbook instance of Goodhart's Law in engineering: when a measure becomes a target, it ceases to be a good measure. This is precisely why the quality of eval construction often matters more to a project's success than which model you choose.

When Rules Stop Working, Nobody Writes It Down

Another pain point raised in these discussions is equally sharp: no one records when a rule stops being valid.

Business rules are dynamic. Regulatory policy changes, shifting market conditions, and internal process iterations all render once-correct practices obsolete. But organizations rarely have mechanisms to track "from what date did this rule stop applying?" The result is a knowledge base that mixes valid and outdated logic indiscriminately — and only the expert who carries the latest context in their head can tell the difference. Which, again, circles the problem back to dependence on expert consensus.

For teams building agents, this means training data carries an inherent risk of temporal decay. A sample labeled as correct today might become a counterexample months later, with no explicit signal to prompt you to update it.

This phenomenon closely maps to the concept of knowledge decay in knowledge management. Organizational knowledge bases, in the absence of active maintenance, see their accuracy decline exponentially over time. In compliance-heavy industries like finance and healthcare, regulatory documents may be updated quarterly, while internal knowledge base synchronization often lags by months or even years.

For machine learning systems, this creates a particular challenge: a model won't proactively tell you it's making decisions based on outdated logic. In software engineering, deprecated dependencies trigger compiler warnings; but in AI training data, temporally decayed samples are format-identical to valid ones — only someone familiar with the latest business context can spot them. This is why some teams are beginning to adopt data versioning tools (such as DVC or LakeFS) and experimenting with attaching business rule version numbers to individual training samples, enabling precise identification of affected sample sets when rules are updated.

Practical Implications

This discussion started from an open-ended question, but it points toward several actionable directions.

Treat disagreement as signal, not noise. When two experts give different answers to the same case, that often marks a genuinely ambiguous, high-risk area in the business. Rather than forcing a single canonical answer, record the disagreement itself and clarify under which conditions each judgment applies.

Build temporal metadata into rules. When crystallizing knowledge, simultaneously record the conditions under which a rule is valid, its applicable scope, and a review cadence. This is more valuable than simply accumulating "correct answers" — and it mitigates the problem of rules quietly expiring without anyone noticing.

Prioritize process design over tool selection. Since the bottleneck isn't in the toolchain, the mechanisms that truly determine project success are: who provides the answers, how disagreements get resolved, and how knowledge is updated over time.

A Question Worth Continuously Asking

The original post ended with a practical question: in your last project, who ultimately gave you the right answers? And what happened when two people disagreed?

There's no standard answer to these questions. But they remind every team working on vertical AI of something important: the hard part of building domain agents is shifting from "can the model learn this" to "can humans articulate it clearly." As large model capabilities advance rapidly, an organization's ability to accumulate, align, and maintain expert knowledge may be the real differentiator — the true moat that separates the leaders from the rest.