- Artificial Intelligence
Why Websites Block AI Agents, and How to Let the Right Ones In
Why Websites Block AI Agents, and How to Let the Right Ones In
Personal AI agents can now browse, book and buy on your behalf, but they keep getting blocked by the websites they visit. That happens for two reasons: some sites block agents on purpose for business or legal reasons, and others block them by accident, because anti-bot defences and "prove you are human" checks cannot tell a customer's legitimate agent from a malicious scraper. The fix is not more blocking. It is agent identity. New standards such as Web Bot Auth let a good agent prove which operator it belongs to, so a site can welcome verified agents, slow down unknown ones, and still keep bad actors out.
The web's bot defences were built to sort people from automation. An agent breaks that, because it is automation working for a real person. Sites now need a way to tell a trustworthy agent from a harmful one.
Some blocks are deliberate: Amazon shut Meta's Muse out of its store, and airlines or listing sites may gate agents behind their terms or a paid licence. Others are accidental, when a "verify you are human" step trips up an agent the retailer was glad to have.
Web Bot Auth, an IETF standard, has agents sign each request so a site can confirm which operator sent it. Cloudflare, Anthropic and OpenAI already run it in production.
Knowing who is knocking is only half of it. Agentic checkout standards let a verified agent buy through a proper channel, with no page scraping and no stored passwords. OpenAI, Stripe, Google, Shopify, Visa and Mastercard are all building toward it.
Keep agents out, strike deals with a chosen few, or open up to any agent you can verify. If you run a website, that decision is now yours to make on purpose.
Turn an agent away and you usually turn away the person it is shopping for, and they tend to buy somewhere else instead.
Agents can act now, and the open web is pushing back
For two years, AI assistants mostly answered questions. In 2026 they started doing things. A new wave of consumer agents, including Meta's Muse, OpenAI's agent features and others, can browse a site, fill in a form, book travel and complete a purchase on a person's behalf, with no coding required by the user.
That capability immediately collided with the web as it is actually built. In September 2026, twelve days after Meta launched Muse, Amazon blocked the agent from browsing or buying on its store, telling shoppers that access by an unauthorised AI agent violated its Conditions of Use. The same month, Shopify did the opposite and wired Muse in for checkout through Shop Pay. A customer whose agent is welcome at one storefront and blocked at the next is left confused, and often blames both the agent and the retailer.
This is the real frontier of agentic AI. Not whether an agent can complete a task, but whether the website on the other end will let it. For any business that sells or serves customers online, that is now a decision you have to make on purpose, because if you do not decide, your anti-bot stack decides for you.
The two kinds of blocking
Agents get blocked for two very different reasons, and telling them apart matters because each one needs a different fix. One is a deliberate business or legal decision. The other is a technical accident that turns away customers a site actually wanted.
A deliberate business decision
Amazon blocked Muse after Meta declined to exclude Amazon from the experience, citing a rule that agents must identify themselves and an allegation that the agent could reach account data. Some airlines point to terms of use that forbid automated access. Yelp allows agents only through a paid data-licensing deal. These are choices about competition, control and revenue, not bugs.
A defence that fires at the wrong target
Walmart partnered with Muse and wants those customers, yet shoppers still hit failures when a human-verification step interrupted the agent and the check failed. The retailer said the blocks were not intentional. This is the more common and more fixable case: a site turning away business it meant to keep, because its anti-bot layer was built for a web without agents.
Why the old trust layer breaks in the agent era
The web's defences were designed to answer one question: is this visitor a human or a bot? CAPTCHAs, the small set of allow-listed crawlers like Googlebot, user-agent strings and IP range checks all exist to separate people from automation and to wave through a handful of known, well-behaved bots.
Personal agents break every assumption in that model. They are automation acting for a real human, so "prove you are human" is the wrong test. There are not a handful of them but potentially thousands, so a hand-maintained allow-list does not scale. And the signals sites have traditionally relied on, the user-agent string and the IP address, are trivially forgeable, so a site genuinely cannot tell whether the request claiming to be a helpful shopping agent is one, or a scraper wearing the same label.
Cloudflare alone now sees AI bot requests in the tens of billions per week. At that scale, treating every automated request as suspect blocks good agents, and treating them as trusted invites abuse. The web needs a way to answer a new question: not "are you human," but "which agent are you, and do I allow that one?"
The fix: agent identity with Web Bot Auth
Web Bot Auth is an emerging IETF standard that lets an automated client prove which operator it belongs to by cryptographically signing each request, instead of relying on a forgeable user-agent string. A site or its CDN verifies the signature against a published directory and learns, with confidence, that a request really is from ChatGPT, Claude, Muse or an unknown party. It already runs in production at Cloudflare, Anthropic and OpenAI, which made it a de facto standard ahead of the finished specification.
Technically, Web Bot Auth builds on HTTP Message Signatures (the IETF's RFC 9421). Each request carries a signature plus headers that point to the operator's identity and signing key, which a site fetches from a published directory and checks. A forged user-agent cannot fake a valid signature, so the site learns the true operator no matter what the request claims to be.
Two points matter for anyone planning around it. First, a verified signature proves who the operator is, not that the request is authorised or well-intentioned. A verified agent is still an agent, and whether it may read pricing, add to a cart or touch an account is a separate policy you set. Second, if you already sit behind a CDN that supports signed agents, such as Cloudflare or Akamai, you can switch verification on there with little custom work, and choose to welcome verified agents while slowing down everything unproven. Supporting it is itself a signal that you want agent traffic, which makes your site meaningfully easier for AI to use than a site that blocks everything.
The commerce layer: letting a verified agent actually buy
Knowing who is knocking is step one. The second question is how an agent completes a transaction without scraping your pages or holding a customer's password. A set of agentic commerce standards has emerged in parallel to answer exactly that, so an agent can discover products, build a cart and check out through a sanctioned path while the merchant stays the system of record.
| Standard | Who is behind it | What it does |
|---|---|---|
| Web Bot Auth | IETF, led by Cloudflare with Google, in production at Anthropic and OpenAI | Verifies which agent operator is sending a web request, before any payment |
| Agentic Commerce Protocol (ACP) | OpenAI and Stripe | A standard checkout an agent can call, with the merchant as system of record |
| Commerce protocol for discovery and checkout | Google and Shopify | Discovery, cart and checkout, used by Shopify's agentic storefronts and Shop Pay |
| AP2, Agent Payments Protocol | Google and partners | Defines the payment message formats passed between agents |
| Visa Trusted Agent Protocol, Mastercard Agent Pay | Visa, Mastercard | Build on agent verification to authenticate shopping agents and tie them to a real user mandate |
The direction of travel is clear. Industry groups are now forming broader norms for agent-to-agent commerce, with retailers, payment firms and agent platforms at the table. For a merchant, the practical takeaway is that you no longer have to choose between an open front door for scrapers and a locked one for customers. There is now a supported middle path: verify the agent, then let it transact through a channel you control.
Block, partner, or open: the choice every business now faces
If your business has a website, agent traffic is already arriving, and you face three real options. Which one fits comes down to your business model.
Block
Keep agents out to protect margins, ad revenue or the direct customer relationship. Defensible when agents would cannibalise your model, but you turn away the customer behind every agent you refuse.
Partner
Allow-list specific agent operators through formal deals, the Walmart and Shopify route. You get reach plus control, but it does not scale to the long tail of agents your customers actually use.
Open via standards
Adopt agent verification and agentic checkout so any verified agent can transact on your terms. The most scalable and future-proof path, and increasingly the default for commerce.
How to make your website agent-ready
Whether you choose to partner or to open up, the engineering work is similar and none of it requires rebuilding your site. This is the checklist we use with clients.
- Decide a policy per surface. Browsing, buying and account actions carry different risk. You can welcome verified agents on product and pricing pages while keeping stricter rules on checkout and login.
- Turn on signed-agent verification at your edge. If you use Cloudflare, Akamai or a similar CDN, enable Web Bot Auth or signed agents there. Allow verified operators, rate-limit the unverified, and block known bad actors.
- Keep human checks out of the agent path. Do not put a blanket "click to prove you are human" step in the critical flow for verified agents. Use risk-based challenges that only trigger on genuinely suspicious traffic.
- Make your content machine-readable. Clean structured data, accurate product and pricing feeds, stable URLs and clearly stated policies help an agent get the facts right. Agents that guess at prices create refunds and complaints.
- Offer a sanctioned checkout. Adopt an agentic commerce path so a verified agent can complete a purchase without scraping pages or storing a customer's credentials.
- Publish clear agent terms. State what agents may and may not do, the way a robots file states crawler rules. Good operators follow stated norms, and clarity reduces accidental blocks.
- Monitor and measure. Track verified agent traffic and the conversions it drives, so your block, partner or open decision is based on your own data rather than headlines.
For businesses in India in particular, agent traffic is not only arriving through browsers. It is arriving through WhatsApp-native assistants, super-app agents and in-app shopping flows, which means the same question applies to conversational surfaces as to your website. The brands that treat agent access as a deliberate design decision now, rather than an anti-bot afterthought, will be the ones agents can actually transact with when consumer adoption scales.
Frequently asked questions
Why do websites block AI agents?
For two reasons. Some sites block agents on purpose, for competitive, legal or data-licensing reasons. Others block them by accident, because anti-bot defences and "prove you are human" checks cannot tell a customer's legitimate agent from a malicious scraper, so the agent is stopped mid-task even when the site wanted the sale.
Why did Amazon block Meta's Muse?
In September 2026, about twelve days after Muse launched, Amazon blocked it from browsing and buying on Amazon.com, citing its Conditions of Use after Meta declined a request to exclude Amazon. Amazon said the agent did not identify itself as an agent on every request and could reach account information. Meta said Muse cannot see passwords or payment details because credentials sit in secure storage.
What is Web Bot Auth?
Web Bot Auth is an emerging IETF standard, built on HTTP Message Signatures (RFC 9421), that lets an automated client sign each request with a verifiable key. A site or CDN checks the signature against a published directory and learns which operator the request belongs to, replacing forgeable user-agent strings. Cloudflare, Anthropic and OpenAI already run it in production.
Does a verified agent automatically get access?
No. Verification proves which operator is knocking. It does not prove the request is authorised or well-intentioned. The site still decides whether that operator may read pricing, add to a cart or touch an account, and whether to allow, rate-limit or block it. Authorisation and intent are separate layers from identity.
What are agentic commerce protocols?
They are standards that let a verified agent complete a purchase through a sanctioned path instead of scraping pages or storing credentials. Examples include the Agentic Commerce Protocol from OpenAI and Stripe, a commerce protocol from Google and Shopify, and agent payment work from Visa and Mastercard. In each case the merchant stays the system of record for the order.
Should my business allow AI agents on its website?
It depends on your model. There are three options: block agents, partner with specific operators, or open up to any verified agent through standards. For most consumer businesses, refusing an agent usually means refusing the shopper behind it, so the momentum is toward verified access rather than blanket blocking.
How do I make my website agent-ready?
Decide a policy per surface, enable signed-agent verification at your CDN, keep human-verification checks out of the critical path for verified agents, publish clean structured data and accurate pricing, offer a sanctioned agentic checkout, state clear agent terms, and monitor the agent traffic and conversions you receive.
Will CAPTCHAs still matter in the agent era?
Yes, against unverified and malicious automation. But a blanket CAPTCHA in the main flow now blocks legitimate agents that a site may want. The better pattern is to verify known agents cryptographically and reserve challenges for genuinely suspicious traffic, rather than treating every automated request as a threat.
Deciding how to handle AI agents on your site?
Auriga IT helps businesses build AI agents and make their websites, storefronts and ERP systems ready for agent traffic, from signed-agent verification to agentic checkout. We will help you choose whether to block, partner or open up, and implement it cleanly.
Published 7 October 2026 by Auriga IT. Agent access standards are moving quickly, so figures and partnerships here are accurate as of early October 2026.
Related content
Auriga: Leveling Up for Enterprise Growth!
Auriga’s journey began in 2010 crafting products for India’s [...]
