AI Agent Vendor Questionnaire: Questions to Send Before You Buy

Use this questionnaire to get written answers from an AI agent vendor on permissions, approvals, logs, data handling, pricing and exit. Send it before the contract stage, and compare the answers side by side. A vendor that cannot answer these clearly has told you something useful.

This template is part of our AI agents for business hub. It was checked against OWASP and vendor documentation on 9 November 2026. It is general information, not legal advice, so have your attorney review the contract and data terms.

How to use it

Send the questions with your own one-paragraph description of the job. Ask vendors to answer "Yes, No or Partly," with a sentence of detail and a link to documentation.

The questions draw on OWASP's Top 10 for Agentic Applications, which OWASP dated 9 December 2025, and its AI Agent Security Cheat Sheet. Neither is a certification, and a good answer here does not make a product safe.

Section 1: What the product is

1. In your product, does the model choose the next step, or does fixed code?
2. Which model or models does it use, and can you change them without telling us?
3. What happens when the model provider changes or retires a model?
4. Which parts are generally available, and which are beta or preview?

The first question comes from the distinction in Anthropic's guide to building agents. Beta status matters because it can change what data terms apply. Anthropic's data retention page says: "Features in beta are generally not covered under the BAA unless explicitly listed as eligible in the feature eligibility table."

Section 2: Permissions and access

The cheat sheet says: "Grant agents the minimum tools required for their specific task."

1. List every tool or system the agent can reach by default.
2. Can we disable individual tools?
3. Can access be limited per user, per customer and per record?
4. Does the agent act with its own identity or the user's?
5. How are API keys and credentials stored, and can they be scoped?
6. Are limits enforced in the systems it connects to, or only in the prompt?

Question 10 matters because of OWASP's Q3 2026 exploit roundup. It says: "Prompt-level instructions alone do not establish a secure boundary." That roundup appears on OWASP's site but says it is not an official OWASP publication.

Section 3: Human approval

1. Which actions can be set to need approval?
2. Can we add our own rules, by action and by amount?
3. Does the approver see exactly what will happen before it happens?
4. Can a person interrupt a run and undo its actions?
5. What is the default setting for a new agent?

The cheat sheet says: "Require explicit approval for high-impact or irreversible actions." Ask about defaults specifically, because they vary. Anthropic's permission policies page, for example, says its built-in toolset defaults to always allow, and MCP toolsets default to always ask.

Section 4: Logs and monitoring

1. Do you log every decision, tool call and outcome?
2. Can we export logs to our own systems?
3. How long are logs kept, and can we set that period?
4. Can we see which source documents produced an answer?
5. What alerts exist for unusual behavior or cost?

Section 5: Data handling

1. What data leaves our environment, and where does it go?
2. Do you use our data to train any model? Where is that stated in the contract?
3. Which subprocessors handle it, and in which countries?
4. What is retained, for how long, and how do we delete it?
5. Do you offer zero data retention, and on which features?
6. If we handle regulated data, will you sign the agreement we need, and which features does it cover?

Anthropic's Managed Agents overview says that product is not currently eligible for Zero Data Retention or HIPAA BAA coverage. Other Claude API features can be, so ask at feature level and have counsel decide what you need.

Section 6: Security testing

1. Do you test for prompt injection, and how often?
2. Can you share a current third-party assessment or report?
3. How do you handle a compromised tool or integration?
4. Which risks on the OWASP agentic list have you tested against?

OWASP's roundup names ten risks. They are Agent Goal Hijack, Tool Misuse and Exploitation, Identity and Privilege Abuse, Agentic Supply Chain Vulnerabilities and Unexpected Code Execution. The other five are Memory and Context Poisoning, Insecure Inter-Agent Communication, Cascading Failures, Human-Agent Trust Exploitation and Rogue Agents.

Do not accept "we are fully protected" as an answer. Ask what was tested and what was found.

Section 7: Quality

1. What accuracy do you measure, on what data, and how recently?
2. Can we run our own test set during the trial?
3. How do you test for regressions when you update the product?
4. How will you tell us when behavior changes?

Section 8: Pricing and limits

1. What exactly is the billing unit, and when is it counted?
2. Are there minimums?
3. What happens to unused credits at the end of a term?
4. Can we cap spending, and what happens at the cap?
5. Does the agent stop at a limit, or does overage bill automatically?
6. How many units did your last comparable customer use per task?

Unused balances are a good example of why this matters. Salesforce's pricing page says: "Unused Flex Credits do not roll over into subsequent subscription terms." Our hidden costs guide covers the rest.

Section 9: Incidents

1. How do you tell us about an incident, and how fast?
2. Who is our contact during one?
3. Can we stop the agent and revoke its access ourselves, immediately?
4. Will you share logs needed for our own notifications?

Section 10: Exit

1. What can we export: instructions, tools, tests, logs, conversation history?
2. In what format, and at what cost?
3. How much notice do you give before retiring a product or feature?
4. How long do you keep our data after we leave, and how do you confirm deletion?

Notice terms are not theoretical. OpenAI's deprecations page shows a 3 June 2026 notice for Agent Builder, with shutdown on 30 November 2026.

How to read the answers

Then test the claims. Ask vendors to show two or three of these answers in a live trial, as described in how to choose an AI agent vendor.

Next step

Delete the questions that do not apply to your job, then send the list to your top two or three vendors. If you would like a builder's reading of the answers, contact our AI agent development team.

Frequently asked questions