Why Forward Deployment Has to Be Forward

The requirements document rested on an assumption so ordinary that nobody stated it: that what software should do could be known before the software existed. Getting it written was miserable — six weeks of workshops, three departments disagreeing about one process, a signature that meant less than it looked like. But the difficulty was political, not fundamental. The knowledge was in the building somewhere, enough time in rooms would extract it, and once extracted it could be handed to people who had never met the customer.

That assumption is what breaks with systems built on models rather than rules, and it breaks more completely than the current conversation admits.

Translation is the closer comparison

Consider what it would mean to specify a translation in advance.

The requirement is easy to write and useless: render this accurately in French, preserving meaning and tone. Nobody disagrees with it and it settles nothing. Everything of consequence lives in decisions that sentence doesn't reach — whether an idiom becomes an idiom or a paraphrase, whether a deliberately awkward line stays awkward, whether a term of art gets the standard rendering or the one readers will understand. You can't list those cases beforehand, because you don't know which ones the text contains until you work through it, and you can't settle them by rule, because the rules underdetermine the answer.

What you can do is judge a candidate translation, line by line, if you know both languages and the subject. So the specification isn't a document written first. It's a series of judgments made against real material by someone competent to make them — and the competence has to be present while the work happens.

The evaluation set is the specification

An evaluation set is a collection of real inputs paired with the answer someone has decided is correct, weighted toward the cases where the correct answer isn't obvious.

Written well it does everything the requirements document was supposed to do, and one thing more. It defines what the system should produce. It settles the edge cases, because you can't include one without deciding it. And it executes, so whether the system meets the specification stops being a matter of opinion.

It's also more honest than what it replaces. A requirements document states intentions. An evaluation set states commitments — this input, this answer, this person, this date. The awkward cases that workshops used to defer as open items can't be deferred here, and the discomfort of settling them is most of the value.

Which is why the work has to be co-located

Authoring one requires two kinds of knowledge that rarely sit in the same head.

You need someone who knows the domain well enough to say what's right when it's genuinely contested — not the textbook answer, the one an experienced practitioner would defend. And you need someone who knows how the system fails, so the set probes where it's weak rather than where it's obviously fine.

Left alone, each produces something useless. The domain expert writes representative examples the system passes easily. The engineer writes a set that measures fluency, or self-consistency, or something else that isn't the thing that matters. Both people, same material, same time.

What a good one looks like

The inputs are real. Synthetic examples test the system against your imagination, which it will pass, because you wrote both sides. Pull from the customer's actual data, including the malformed records and the ones somebody entered wrong in 2019.

Hard cases are over-represented. A set that mirrors real-world frequency is mostly easy cases, and a system can score well on it while failing every instance that matters. You aren't sampling the world. You're sampling the places where being wrong is expensive.

Some cases have no answer. If every question in the set has one, the system learns that every question has one — and you've trained away the ability to say I can't determine this from what I was given, usually the most valuable thing it can say.

Answers carry names and dates. In six months somebody will dispute a case, and the useful question is who decided it and what they knew then. An unattributed answer key is a set of assertions.

Some of it is held back. Cases used for iteration stop measuring anything, because you've been tuning against them.

It grows. Every production failure that surprises you becomes a case. That ratchet is the mechanism by which the system stops repeating itself.

Making sure one actually exists

The hard part isn't knowing this. It's that evaluation sets are nobody's favorite work, they have no demo, and they can always be done next sprint.

Give it a date and an owner. Work that happens "during" the build happens after the build, which is to say never.

Put the domain expert's hours in writing. This is what actually blocks. The person who can settle the contested cases is the customer's busiest employee, and nobody has committed their time. Four hours from them is worth four weeks of engineering, and it has to be negotiated up front, while goodwill is highest.

Write the threshold before you run anything. Once results exist, a number that would have been disappointing on Monday reads as encouraging on Friday, and the reasoning feels sound at every step. Committing in advance costs an hour. Almost everyone agrees it's correct; almost nobody does it, because a stated threshold means the project can fail.

Review the set, not the demo. The demo shows what the system does when things go well, which you could have guessed. The set shows what the team believes correct means, and whether they've thought about the cases that will hurt. For a manager of a function that produces almost no other reviewable artifacts, it's the highest-leverage half hour available.

Where this argument is weak

Evaluation sets go stale without announcing it. They encode one moment's understanding of what matters; the inputs shift, and the set keeps returning a comfortable number while measuring a question nobody is asking.

They're gameable in a way that feels like progress. Optimize against a fixed set long enough and you've built to the test. Every field that adopted a standard benchmark has watched this happen, and the only defenses are holding cases back, rotating them, and staying suspicious of a rising number.

And they're usually too small to carry the confidence placed in them. Thirty cases is a sanity check, not a measurement.

None of this applies everywhere, either. Plenty of systems can still be specified up front, and wrapping an ordinary integration in this apparatus is just a slower way to build it. The argument holds where behavior is statistical and the rare cases are the expensive ones.

Why the engineer is there

The mythology of forward deployment is relationships and responsiveness — flying out, sitting with the customer, turning something around over a weekend. Those are real. They are not the reason.

The reason is that the specification for this class of system can't be produced anywhere else. It needs the customer's actual data, which doesn't travel, and the customer's expert, who won't. It needs someone who knows how the system fails, in the room while those judgments get made. Neither party can write their half and mail it.

That's what forward means. Not near the customer. In the only place the specification can be written.