How to Block AI Bots Without Blocking Google (or Your Google Merchant Center Listings)

You added a few lines to robots.txt to keep AI bots out, or an app did it for you, and now Merchant Center says it can’t check your product pages. Google’s help page for that issue doesn’t soften it: These products will remain disapproved and stop showing up in your Shopping ads and free product listings until we are able to crawl the landing page. (How to fix: Unable to check product pages.)

You can block AI bots without blocking Google. Each crawler follows its own group of rules in robots.txt, so the safe way is to name the AI bots you want out and leave Google’s crawlers alone. Google only shows your products if it trusts your store, and it can’t check pages it isn’t allowed to read.

Which bots does Google need, and which are AI bots?

Google’s crawlers: keep them in

Google lists its crawlers on two pages: common crawlers and special-case crawlers. Four matter for a store:

• Googlebot crawls for Google Search.
• Googlebot-Image fetches images. Google’s Merchant Center fix page asks you to allow both Googlebot (for landing pages) and Googlebot-Image (for images).
• Storebot-Google is Google’s shopping crawler. Google says Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping. Google’s StoreBot page says it reads product, cart and checkout pages for prices, shipping and availability, and adds: It allows Google to verify information you share through Merchant Center, and helps you keep your data more accurate and up to date.
• AdsBot-Google checks your pages for Google Ads. It plays by a different rule: AdsBot ignores the global robots.txt user agent ( * ) with the ad publisher’s permission. A User-agent: * rule never reaches it. Only a group that names it can block it.

Google-Extended looks like a bot but isn’t one. It’s a robots.txt token for whether Google may use your content for Gemini. Google says Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search. Blocking it is a choice about Gemini, not about Search or Shopping.

AI crawlers: your choice

• OpenAI (crawler page) runs three. GPTBot is for training: Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models. OAI-SearchBot is for ChatGPT’s search features; OpenAI says Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. ChatGPT-User visits a page when a user asks, and OpenAI says robots.txt rules may not apply to it.
• Perplexity (crawler page) runs PerplexityBot for its search results, and says of it: It is not used to crawl content for AI foundation models. Perplexity-User fetches pages for user questions and, Perplexity says, generally ignores robots.txt.
• Microsoft (Bing’s crawler list): Bingbot is our standard crawler and handles most of our crawling needs each day. It’s Bing’s search crawler, so blocking it means leaving Bing. Bing’s page lists AdIdxBot as the crawler for its ads.

For more crawler names and how to see their visits, see how to track AI crawler visits on your Shopify store.

Does blocking AI bots remove you from Google Shopping?

No. A group that names GPTBot, OAI-SearchBot or PerplexityBot is read only by that bot. Google’s crawlers pick their own group. Google’s robots.txt rules say the crawler uses the group with the most specific name that matches it, and User agent specific groups and global groups ( * ) are not combined.

What does hurt Merchant Center is any rule that reaches Googlebot, Googlebot-Image, Storebot-Google or AdsBot-Google: a group that names them, or (except for AdsBot) the * group they fall back to. Our reading: the AI side is a decision about AI tools only. The Google side is where your listings and ads live.

How to block AI bots without blocking Google: a safe robots.txt

Here is a pattern that leaves Google alone. The * group stands for whatever rules your platform already writes; keep those as they are.

# Your platform's existing rules (unchanged)
User-agent: *
Disallow: /cart
Disallow: /checkout

# AI training: opt out
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

What each part does:

1. User-agent: * is the group for every crawler that has no group of its own. Googlebot, Googlebot-Image and Storebot-Google follow it here, so they can read every product page.
2. User-agent: GPTBot / Disallow: / asks OpenAI not to use your pages for training. ChatGPT search isn’t affected; that’s OAI-SearchBot.
3. User-agent: Google-Extended / Disallow: / opts out of Gemini use. Google Search and Shopping aren’t affected.
4. No line names Googlebot, Storebot-Google or AdsBot-Google. That’s on purpose: naming them is how mistakes start.
5. OAI-SearchBot, PerplexityBot and Bingbot aren’t named either, so they follow the * group. If you want those out too, give each its own group with Disallow: /, knowing it costs you that tool’s search results.

Add other training crawlers the same way, by the exact name their company publishes.

Which robots.txt mistakes lock Google out?

1. “Block everything, allow Googlebot.”

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

This looks safe and isn’t. Google’s own example says Storebot-Google follows group 2, because there is no specific Storebot-Google group. Group 2 in that example is *. So in this file Storebot-Google, Google’s shopping crawler, is blocked from the whole store, even though Googlebot gets in.

2. A group that names AdsBot. Because AdsBot skips *, it’s only blocked when someone names it. Some “block all bots” lists and robots.txt editors do. Check for AdsBot-Google or AdsBot-Google-Mobile followed by Disallow: /.

3. Blocking product folders for everyone. A Disallow: /products/ or Disallow: /collections/ under *, meant to slow down bots, also stops Googlebot and Storebot-Google.

4. A firewall that does it for you. Robots.txt isn’t the only gate. Cloudflare’s Block AI Bots docs describe its older setting like this: This setting blocks verified bots that are classified as crawling for the purpose of AI training, as well as a number of unverified bots that behave similarly. The same page says that on 15 September 2026, new domains will get new defaults: training and agent bots blocked on pages that show ads, and search crawlers allowed. Our reading: a setting that challenges or rate-limits Google’s crawlers can cause the same Merchant Center issue as a robots.txt rule. On its page for local listings, Google says If you use a firewall or a CDN like Cloudflare, ensure that Google’s crawlers are whitelisted and your server is not returning 403 or 404 errors only to automated bots. It also says We don’t recommend the static allowlisting of IP addresses. and points to a reverse DNS lookup instead (How to fix: Google StoreBot crawler can’t access your in-store product page).

How do I check that Google can still crawl my store?

1. Open https://yourstore.com/robots.txt. Read every User-agent line, not just the top of the file.
2. Find each Google crawler’s group. Googlebot and Storebot-Google follow a group that names them, or else *. AdsBot-Google follows only a group that names it. Look in each for Disallow: /, or for disallowed product, collection or image paths.
3. Use Search Console. The robots.txt report shows the file Google last fetched and any errors. Run a product page through the URL Inspection tool, which Google’s Merchant Center fix page also suggests.
4. Look in Merchant Center. Under Products → Needs attention, look for crawl issues such as “Unable to check product pages”. Our guide to “Product page unavailable” covers the related landing page errors.
5. Check your firewall or CDN. Review bot and AI settings, and any rate limits, for rules that catch Google’s crawlers.
6. Re-check after every change. Installing an app, changing theme, adding a security plugin or moving DNS can each rewrite these rules.

ShieldKit’s free store check reads a Shopify store’s public pages: whether the storefront is open to visitors, and whether product pages carry product data Google can read. Reading each crawler’s robots.txt group is still a job for the steps above.

On Shopify, WooCommerce and other platforms

Shopify

Shopify writes robots.txt for you. To change it, you add a robots.txt.liquid template to your theme. Shopify’s help page is frank about the risk: Incorrect use of the feature can result in loss of all traffic.

• Keep Shopify’s default rules. Shopify’s developer docs show how to add your own groups after the Liquid that prints the default rules, rather than replacing the file.
• Leave the adsbot-google group alone. On the Shopify stores we looked at, the default file already has a separate adsbot-google group that allows product and collection pages and keeps checkout out. That’s Shopify handling AdsBot skipping *.
• Check what apps changed. If an AI or SEO app offers to edit robots.txt, read the file after you switch the feature on.
• AI channels are separate. Shopify’s help page says blocking AI crawlers affects open-web discovery only, and doesn’t stop Shopify Catalog sending your product data to AI channels you’ve turned on. For that side, see will my Shopify store appear in ChatGPT Shopping.
• Proxies: the same page says Shopify doesn’t recommend a proxy in front of Shopify and can’t support one.

WooCommerce and WordPress

• Find who writes robots.txt. WordPress serves its own virtual file, a physical robots.txt in the site’s root folder replaces it, and SEO plugins often have a robots.txt editor. Open the live file to see which one won.
• AI-blocking plugins: check the groups they add, and make sure none names Googlebot, Storebot-Google or AdsBot-Google.
• Security and caching plugins can block or rate-limit by user agent. Review their bot settings.

Wix, Squarespace and other platforms

Most hosted builders write robots.txt for you, with an SEO setting to edit it and sometimes a switch for AI crawlers. Whatever the setting says, open /robots.txt afterwards and read the groups yourself.

What we see in real cases

• An AI app stopped Google Ads. Two app reviews from February and March 2026 describe an AI feed app whose robots.txt editor blocked Google’s ad crawler. Both merchants’ Google Ads stopped. The developer later said the editor was fixed.
• Blocking AI crawlers, then a warning from Google. In January 2026 a site owner used a plugin to block AI crawlers, and Google then reported that the robots.txt blocked normal indexing. They added Google allow rules by hand; the thread closed without an answer.
• Firewalls in front of Shopify. In February and May 2026, Shopify merchants with Cloudflare in front of their store reported Google’s crawlers getting “too many requests” errors and products flagged as not crawlable. In July 2026, a WooCommerce store reported its hosting’s Cloudflare layer blocking Merchant Center’s crawls.
• Blocked without knowing. In one Shopify thread, a store found Cloudflare’s managed robots.txt had been blocking AI search bots, a “one-click own-goal”, as the thread put it. Another found robots.txt was blocking their policy pages during a Merchant Center review.
• Why people block at all. Bots hammering wishlist and filter URLs pushed one WooCommerce store’s server to full CPU, and robots.txt rules didn’t stop them. And a Shopify merchant was offered a four-figure service to “unblock” AI crawlers, and suspected a scare tactic. Our reading: reading your own robots.txt takes a few minutes and costs nothing.

FAQ

Does blocking GPTBot affect Google Shopping?

No. Google’s crawlers don’t read a group written for GPTBot, so a GPTBot rule doesn’t change what Googlebot, Storebot-Google or AdsBot-Google may crawl. What hurts Google Shopping is a rule that reaches Google’s own crawlers, often through the User-agent: * group.

Should I block Google-Extended?

That’s your call. Google says Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search. It’s a token for whether Google may use your content for Gemini, not a crawler that visits your store.

Why does AdsBot-Google need its own group?

Google says AdsBot ignores the global robots.txt user agent ( * ) with the ad publisher’s permission. So rules in your User-agent: * group don’t apply to it, and only a group that names AdsBot-Google can block it.

Is robots.txt enough to stop AI bots?

No. Robots.txt is a request, not a lock. OpenAI and Perplexity both say their user-requested fetchers may not follow it, and bots that ignore it need a firewall or bot-management rule, which must still let Google’s crawlers through.