Ask a team how their LLM feature is doing and you learn a lot from the shape of the answer. “It feels pretty good” means nobody knows. A number, a dataset, and a threshold mean the feature is engineered.
Start with the eval, not the prompt
We build the evaluation harness before we tune anything: a set of real cases, a definition of correct, and an automated way to score changes. It feels slower for the first week. It is dramatically faster every week after, because every prompt change, model swap, and retrieval tweak gets judged by the same yardstick instead of by whoever looked at the output last.
Evaluation is also what makes model upgrades boring — in the best sense. When a new model ships, we rerun the suite, read the diff, and know within an hour whether to migrate. That is what trust in an LLM feature actually looks like: not confidence in a model, but confidence in your measurements.