This article is part of our AI Agents for Business hub. Want help applying it? Talk to eSEOspace.
Choose an AI agent vendor by testing it on one real job, using your own data and your production limits. A good demo proves very little. What matters is what the product does when your data is messy and an action could go wrong.
This guide is part of our AI agents for business hub and was checked against primary sources on 9 November 2026. It is general information, not legal advice.
If you are hiring someone to build a custom agent, read how to hire an AI agent developer instead. This page is about buying a product.
Key Takeaways
- "Agent washing" means selling a chatbot or a fixed workflow as if it were an agent.
- Anthropic's definition gives you a test: does the model choose its own steps and tools?
- A workflow is not worse than an agent, and for fixed steps it can be the better purchase.
- Ask for a trial on your own data, with permissions and logs turned on.
- Read the billing meter and the exit terms before you compare prices.
What agent washing is
Agent washing is a plain term for marketing an ordinary product as an autonomous agent. The product may be a chatbot with a new name, or a fixed sequence of steps with a language model in one of them.
There is a simple test. Anthropic's guide to building agents says: "Workflows are systems where LLMs and tools are orchestrated through predefined code paths." In its agents, by contrast, the model directs its own process and tool use.
Anthropic sells AI models, but the distinction is useful whoever makes the product. Ask the vendor: "Who decides the next step, the model or your code?"
A vendor describing a workflow accurately is not doing anything wrong. Anthropic's advice is to find the simplest solution possible and add complexity only when needed. The problem is paying agent prices, or accepting agent risk, for something else.
Six signs of a relabeled product
| What you hear | What to ask |
|---|---|
| "It's fully autonomous" | Which actions run without a person, and which can't? |
| "It learns your business" | From what data, and how do you correct it? |
| "It connects to everything" | Show the connection to our system, with its permissions |
| "It works out of the box" | What did your last customer have to set up? |
| "It resolves most tickets" | What counts as resolved, and who decided? |
| "It's safe and private" | Show me the written data terms and the log of actions |
Treat any accuracy claim the same way. Ask what was measured, on whose data, and when. A single number tells you little without the method.
How to run the evaluation
1. Write down the one job
Name the job, the systems it touches, and what a mistake would cost. If you can't describe it in a paragraph, you are not ready to compare vendors.
2. Build a small test set
Anthropic's guide to agent evals says: "In reality, 20-50 simple tasks drawn from real failures is a great start." Pull yours from old tickets, emails or orders, and write down the correct result for each.
Include awkward cases: missing data, angry customers, requests the agent should refuse. Give every vendor the same set.
3. Trial it with real limits
Run the trial with the permissions you'd use in production. A demo with admin access tells you nothing about the product you'd operate.
OWASP's page on Excessive Agency names three root causes: excessive functionality, excessive permissions and excessive autonomy. It lists mitigations including minimal tools, least privilege and human approval for high-impact actions. Ask each vendor to show how its product handles all three.
4. Check the controls
OWASP's AI Agent Security Cheat Sheet gives you concrete things to look for. It says: "Log all agent decisions, tool calls, and outcomes." It also says: "Require explicit approval for high-impact or irreversible actions."
So ask to see a real log from your trial. Then ask to see an action paused for approval. Those two screens will tell you more than a sales deck.
5. Read the meter
Vendors charge in units that are not comparable. On 9 November 2026, three vendors looked like this:
| Vendor | Unit | Listed price |
|---|---|---|
| Salesforce Agentforce | Flex Credits | $500 per 100,000 credits, with Agentforce actions at 20 credits |
| Intercom Fin | Outcome | $0.99 per outcome |
| Zendesk | Automated resolution | $1.50 committed, $2.00 pay-as-you-go |
These do not measure the same thing. Our Fin vs Zendesk comparison shows why, and the hidden costs of AI agents covers the rest of the bill.
6. Plan how you would leave
Ask what you can export, in what format, and how much notice you get. Our vendor questionnaire puts this in writing.
Vendor changes do happen. OpenAI's deprecations page shows Agent Builder scheduled to shut down on 30 November 2026.
A note on vendor claims
Marketing is not exempt from the law just because it mentions AI. The Federal Trade Commission announced a crackdown on deceptive AI claims in September 2024. Its chair at the time said: "The FTC's enforcement actions make clear that there is no AI exemption from the laws on the books."
That announcement was not about agents specifically. Ask your attorney if you think a vendor misled you. The practical lesson for buyers is to get promises in writing.
A simple scorecard
Score each vendor from 0 to 2 on each line, using what you saw in the trial.
| Criterion | 0 | 1 | 2 |
|---|---|---|---|
| Handles your test set | Fails many | Mixed | Reliable on your cases |
| Who chooses the steps | Unclear | Fixed, and honest about it | Clearly described |
| Permissions | All or nothing | Partly limited | Limited per tool and per user |
| Human approval | None | Some actions | Any action you choose |
| Logs | None | Summaries | Every decision and tool call |
| Pricing | Unclear | Clear unit, unclear volume | Clear unit, caps and alerts |
| Exit | None | Partial export | Full export, in writing |
Add up the scores, but read the zeros first. A zero on approvals or logs should end the conversation for any agent that can take action.
Next step
Pick the job and build the test set before your next vendor call. If you already have a pilot that is unreliable or too expensive, our AI agent development team reviews its prompts, tools, permissions and costs. We will tell you what to fix or whether to stop.
Frequently asked questions
What is agent washing?
It means selling a chatbot or fixed workflow as an autonomous agent. The phrase is a description, not a legal term. The test is whether the model chooses its own steps and tools.
Is a workflow worse than an agent?
No. For steps that do not change, a workflow can be cheaper and easier to check. Anthropic recommends the simplest solution that works.
How many test cases do I need?
Anthropic's guide says 20 to 50 simple tasks drawn from real failures is a great start. Use the same set for every vendor, so that results are comparable.
Should I trust vendor accuracy numbers?
Treat them as claims until you can see the test. Ask what was measured, on what data, and how a resolved or successful case was defined.
What should end the evaluation immediately?
For an agent that takes actions, stop if there is no log of what it did or no way to require approval. Stop too if the vendor will not give you written data terms.
