How do you know whether an AI agent got better or worse after each change, and why should you run each test case several times?
Suggested answer
The interviewer wants to know that you have a method for measuring how an agent changes, instead of reading a few answers and concluding that it "looks fine".
A good answer starts with the idea of an eval (an evaluation set): a fixed list of test messages that you run again after every change, such as editing the prompt, switching the model or adding a tool. It's like a unit test, except that an agent has some randomness in it, so there are two decisions of its own: what to score and how many times to run.
On scoring, avoid comparing the answer word for word, because the same idea can be expressed in many ways and the comparison would go red for no reason. Instead, score things you can check on the "trace" of a run: did the agent call the right tool with the right arguments, did it hand over to a person when it should, does the answer contain something required or something forbidden (leaking another person's data, promising a refund). Scoring with code is fast, cheap and gives the same result every time. For things code can't score, such as tone, you can use another model as a judge, or a human grader, but the judge's reliability needs to be checked first.
On how many times to run, a real model can answer differently between two runs, so each case needs to be run several times and the pass rate counted. A rare failure, say one in ten, won't show up if you only run each case five times. Each run is a model call, which means money and time, so split the effort by how much harm a failure causes: cases about security and refunds must pass every time, while everyday cases can allow a lower threshold.
A bonus point: mention turning every failure a customer reports into a new test case, and recognize that an eval only measures what you thought of testing.