Agent Washing: How to Tell Useful Automation from a Better Sales Pitch

A supplier shows you an assistant that understands a complicated request, explains its plan, and announces that the job is done. The demonstration feels impressive. The important question comes afterward: did anything correct and authorized actually happen in the business system?

That gap is where agent washing becomes expensive. The term describes presenting an ordinary chatbot, scripted integration, or incomplete prototype as a broadly autonomous worker. A simpler system is not inherently a bad purchase. Misunderstanding what you are purchasing is the problem.

Start with the work, not the label

There is no universally enforced commercial definition of an AI agent. Anthropic distinguishes predefined workflows from agents that dynamically choose their process and tools. That architectural distinction is useful, but it is not a procurement scorecard by itself.

A fixed invoice workflow can be more reliable and economical than an agent that improvises every step. Conversely, a research task may genuinely benefit from a system choosing which sources to investigate next. Ask why flexibility is necessary for your task, rather than treating flexibility as proof of sophistication.

Define the completed outcome in business language. A customer record updated with verified information is different from a proposed update. A purchase request prepared for approval is different from a purchase order issued. Your acceptance criteria should preserve those distinctions.

An illustrative purchasing example

Imagine a facilities company considering an agent to handle replacement-part requests. The input is an email containing a machine identifier, a photograph, and a description of a fault. The desired output is a prepared requisition, not an unrestricted purchase.

The system reads the approved equipment register, identifies candidate parts, and attaches the relevant maintenance record. Conventional validation checks the asset identifier, current supplier status, available budget, and required approval level. An authorized supervisor reviews substitutions before a requisition becomes an order.

If the asset identifier cannot be resolved, the system asks for clarification or places the item in an exception queue. It must not confidently select a similar machine. If the supplier API times out, it records the pending state rather than claiming success or placing the order twice.

In a credible demonstration, you can inspect the original request, the evidence retrieved, the proposed part, the approval, and the resulting record in the purchasing system. A polished explanation without those artifacts is not equivalent evidence.

Ask for the uncomfortable demonstration

Happy-path examples conceal the most important limitations. Bring a small set of representative, permission-cleared test cases that includes ambiguity, missing records, conflicting instructions, and a downstream failure. Use a sandbox rather than live purchasing authority.

• Ask the vendor to show the exact actions available to the model and the permissions behind each one.
• Include a document containing instructions that should not override your business rules.
• Change a source record between preparation and approval. Does the system detect that its proposal is stale?
• Interrupt execution after a write. Can it determine whether the action succeeded before retrying?
• Ask who investigates a failed case and what information that person receives.

These tests should be proportionate to the task. A draft-writing assistant does not need the same controls as a payment workflow. But a payment workflow should never be approved using evidence that only demonstrates drafting.

Separate model skill from operating authority

The model may be good at extracting a request, comparing documents, or explaining an exception. None of those capabilities grants permission to spend money or change a customer entitlement. The application must enforce the permitted action against the actual user, tenant, record, and approval state.

OWASP identifies excessive permissions and autonomy as a distinct AI security risk. A sentence in a prompt saying “never exceed the budget” is not a substitute for a server rejecting an over-budget transaction.

Ask whether the vendor can narrow the scope without redesigning the product. Can you start read-only? Can one action be disabled? Can access be revoked promptly? A useful system should have an intelligible boundary, not merely a reassuring personality.

Buy evidence of completion

Assess accepted outcomes, exception handling, supervision time, integration effort, and total operating cost. A low error rate on an easy test set may tell you little about the cases your staff actually struggle with. Keep a held-out set of realistic cases the vendor has not optimized against.

Require a written account of what is supported, what needs human handling, what changes invalidate the evaluation, and how you exit the arrangement. If the supplier cannot explain the system without marketing language, you may be inheriting that uncertainty.

Ask for the completion record to be generated by the destination system, not only by the agent's own activity log. Those records can disagree when a remote request fails. The demonstration should show how that disagreement is detected and who reconciles it before the task is counted as accepted.

The best answer to agent washing is not cynicism about AI. It is a better purchasing question: show me the work completed correctly, within its authority, including what happens when it cannot finish.