The AI Demo Passes Until the Customer Changes One Detail
Disclaimer: Sponsored content prepared with SourceX
The customer asks for a refund. The agent replies politely, finds the order and follows the familiar path. The demo looks ready.
Now change one detail: the refund needs an approval that has not been given.
The wording can stay excellent while the action becomes wrong. If the test only grades the answer, the failure may remain invisible.
This hypothetical example points to a practical problem in AI agent evaluation. Business work depends on the information, permissions and system state at a particular moment. Real operational histories can help a team discover which moments its showcase never tested.
A convincing answer can hide the wrong action
An agent that drafts a response and an agent that changes a customer account have different jobs. For an agent allowed to use tools, evaluation needs to inspect what it attempted and what actually changed.
Illustration — A good answer and a correct state are separate checks. Hypothetical refund workflow. This illustration reports no measured model performance.
In the refund example, an appropriate action might be to request authorization, explain the next step and leave the payment unchanged. A confident confirmation message would be a poor result if the required approval never existed.
Anthropic’s engineering guide to agent evaluations distinguishes the transcript of an attempt from its outcome in the environment. That distinction is useful here: saying that a task was completed and establishing the expected system state are separate checks.
Rebuild the case at the moment of decision
Begin with the point at which the agent must choose its next step. Reconstruct only the information it could legitimately know then.
Illustration — The case as the agent could know it. Illustrative case definition. The approved source records determine what can be reconstructed.
- The customer’s request and the relevant account state.
- The policy version in force at that moment.
- The tools and permissions available to the agent.
- Any approval already granted, refused or still pending.
- The expected action and the evidence used to score the result.
Keep the ending out of the beginning. A later resolution can help a reviewer judge an earlier decision, but it should not appear in the prompt as information the agent already had. Otherwise, the test may reward access to the answer rather than the ability to make the decision.
Preserve the relationships among the request, the action and the later outcome. If the source history cannot establish a relationship, document the gap instead of quietly filling it in.
Change one condition and keep the test honest
A useful development exercise is to hold the basic request constant while changing one documented condition. The following is an original, hypothetical test plan, not an industry benchmark:
Illustration — Five ways to change the same refund request. Original hypothetical test plan. Expected behavior depends on the organization’s actual policies.
- Approval is present: follow the approved refund procedure.
- Approval is missing: request it before taking the restricted action.
- The policy version is unavailable: surface the uncertainty for review.
- A prior refund already exists: check the state before creating another.
- The payment tool returns a failure: report the failure and avoid claiming success.
Write the permitted behavior for each case before running it. A different organization may have a different policy, so the expected result must come from its agreed task definition.
These variants can expose a test suite that only recognizes the familiar success path. They do not replace representative coverage of the rest of the product’s workflows.
Score the action and the ending separately
Keep a readable record of the agent’s tool use, the permission checks and the resulting state. A correct sentence should not compensate for an unauthorized transaction.
Illustration — Check the request through to the final state. A development workflow, not a claim that any one score proves production readiness.
For the approval-missing case, a reviewer could check that no refund command was executed, that the case was routed to an appropriate approver and that the customer was told what happens next. For the tool-failure case, the system should not report a completed refund merely because the response was well written.
Some checks can be mechanical. Others need a reviewer who understands the task. Repeat important cases when output variability matters, and record the model, instructions, tools and policy version used. A change in any of those can change what a result means.
Keep the criteria visible. A single pass percentage is difficult to interpret if it hides which actions were allowed, which cases were missing and how uncertainty was handled.
Useful history still needs permission and preparation
Historical support cases can suggest realistic tasks, but access to the help desk does not automatically authorize an external training or evaluation use.
Illustration — Define the proposed use before moving the records. A company approves a defined scope. Access alone does not create an external AI license.
Before sharing records, establish the proposed purpose, the relevant rights and the company’s approval process. Customer information, employee details, third-party material and confidential attachments can change what is feasible. Some cases may need to be excluded entirely.
Preparation should preserve the meaning needed for the agreed task while reducing exposure. Removing names alone does not establish that a record is anonymous or suitable for reuse. The company and its reviewers still need to assess the actual material.
Look for the history before inventing every example
When an established business has years of documented customer work, its operating history may reveal useful edge cases before a development team tries to imagine them all. That is an assessment question, rather than a guarantee that an archive will improve a model.
Illustration — Keep the ending out of the beginning. The task determines the appropriate separation of development examples and evaluation cases.
SourceX pays qualifying businesses to license approved operational data for AI training and evaluation. It manages the agreed licensing process, preparation, delivery and payout. An interested company can begin with a descriptive fit check before any records are requested; eligibility, rights, quality and buyer fit still apply.
For the next demo review, change one detail that should change the action. Ask what the agent could know, what it was allowed to do and what the system proves it did.
A showcase tells you what worked in the showcase. A useful test tells you where the behavior changes.
About the Founder
Noah Loul is the founder of SourceX, helping businesses monetize their proprietary data autonomously for AI training and development. His ambition is to build the world’s largest data infrastructure, powering today’s AI and tomorrow’s superintelligence.
