Google Can’t Reach Your Product Pages: Cloudflare, Hosts and “Product Page Unavailable” in Google Merchant Center

Your product pages open on your laptop and your phone, yet Google Merchant Center has disapproved them as “Product page unavailable”. Google is clear about the cost: These products will stop showing in ads and free listings until Google can access the landing page. (Google: How to fix: Product page unavailable).

The page working for you doesn’t mean it works for Google. Google sends its own crawlers, and your robots.txt, firewall, host or store settings can turn them away while letting you in. Below: which crawlers need access, what blocks them, and how to test what Google gets.

What does “Product page unavailable” in Google Merchant Center mean?

It means Google tried to open the page your product links to and failed. Google says We routinely crawl your landing pages to ensure that they work properly. Its fix page lists the common crawl errors:

• Page not found (404). The link in your product data points to a page that doesn’t exist.
• Too many redirects. Google’s rule: make sure your landing page URL doesn’t have more than 2 redirects.
• Couldn’t connect or HTTP 5xx. Google’s fix: To fix this error, make sure that your website is always available.
• Can’t access your robots.txt file. The product page was unavailable for Google to check because your online store’s robots.txt file couldn’t be reached.
• Hostname not resolvable, a DNS problem, and URL protocol problems, such as a link on a domain other than the one you verified.

Google’s Product crawl issues page adds a page that requires a login, any HTTP 4xx or 5xx answer, and a slow server: The server didn’t respond in time, so the crawl for this product stopped.

Merchants also post it as “Unavailable desktop landing page”, “Unavailable mobile landing page” or a “client 4xx error”.

Why does the page load for you but not for Google?

Because Google’s crawler is a different visitor, and Google’s bar is every visitor: To resolve this issue, identify and fix the error to make sure your landing page is accessible from any device, location, or browser. In our reading of Google’s pages and merchants’ reports, these are the usual reasons:

1. It comes from somewhere else. On the same Product crawl issues page, Google says sites may sit behind a firewall or geo-restrictions that block its crawlers, and, at the time of writing: If you restrict specific regions, you must allow crawling from the US because Google often operates from there. A store that blocks or redirects visitors by country can fail Google even when it never fails you.
2. Bot settings challenge it. Firewalls and CDNs answer suspected bots with a challenge page a person can pass and a crawler can’t. Cloudflare’s Bot Fight Mode is one example.
3. Rate limits answer “too many requests”. Google crawls many product pages in a short time. A host or firewall limit can answer with HTTP 429, and Google says Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error. (Google: How HTTP status codes affect Google’s crawlers).
4. A password or “coming soon” page. You’re signed in to your store admin, so you see the real store. Google sees the lock screen.
5. robots.txt. Either it blocks a Google crawler, or the file itself can’t be fetched, which Google lists as a crawl error on its own.

Which Google crawlers does Merchant Center use?

When this post was written, Google’s Product crawl issues page said Google uses AdsBot-Google for ads and Googlebot for organic search. It warned: Even if one is allowed, the other might be blocked.

Google’s crawler documentation names a third one for Shopping. Of Storebot-Google it says Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping). (Google: Google’s common crawlers).

AdsBot follows different robots.txt rules. Google says AdsBot ignores the global robots.txt user agent (*) with the ad publisher’s permission. (Google: Google’s special-case crawlers). So a User-agent: * group doesn’t apply to AdsBot, but a group that names it does. Rules like these, added by a theme, an app or a plugin, are what to look for:

User-agent: AdsBot-Google
Disallow: /

User-agent: Storebot-Google
Disallow: /products/

Our reading: don’t block Googlebot, AdsBot-Google or Storebot-Google from your product pages, in robots.txt or anywhere else. A firewall rule that lets one through can still block the others.

Anyone can claim to be Googlebot, so allowing a name alone isn’t enough for a firewall. Google says You can verify if a request to your server really is from Google. Its method is a reverse DNS lookup on the visitor’s IP address, then a forward lookup (Google: Verify requests from Google crawlers and fetchers). Many firewall tools have a setting that does this for known search engine crawlers.

Update, 28 October 2026: Google has since rewritten the Product crawl issues page. It no longer names AdsBot-Google. It now asks you to ensure that your robots.txt file allows both user-agents “Googlebot” (used for landing pages) and “Googlebot-image” (used for images) to crawl your full site. The firewall line now says You must allow crawling from the US because Google often operates from there. and asks hosts to allow the Googlebot user agent. Our advice above doesn’t change: block none of Google’s crawlers, and add Googlebot-Image to the list.

How do you find what’s blocking Google?

1. Read the issue in Merchant Center. Open Products, then the Needs attention tab, and filter on the issue. Download the list of affected products. Open one and note the exact error: a 404, a 5xx, robots.txt and a timeout each point somewhere different.
2. Look for a pattern. All products or a few? Did it start after a new app, a security plugin, a Cloudflare change or a move to a new host? Every product failing at once points at the whole site, not one page.
3. Open a product page as a stranger. Use a private window, signed out of your store, and a phone on mobile data. Look for a password page, a “coming soon” page, a “checking your browser” page or a redirect to another country’s store.
4. Open your robots.txt. Go to yourstore.com/robots.txt. It must load, and no group for Googlebot, AdsBot-Google or Storebot-Google may block your product pages.
5. Test with Google’s own tool. Google’s fix page says You can test your current configuration with the URL Inspection Tool. In Search Console, inspect a product URL and run the live test. Search Console explains: To test the current version of the page as Google would see it, select the Live Test button on the page. (Google: URL Inspection tool). One caveat: the test runs as Google-InspectionTool, not as Storebot-Google or AdsBot-Google (Google’s common crawlers). Our reading: a pass is a good sign, not the final word, because a firewall can treat them differently.
6. Read your firewall’s log. On Cloudflare, open Security → Analytics and the Events tab: requests Bot Fight Mode challenged are labelled with its name. Filter for user agents containing “Google” and look for challenges, blocks, 403s and 429s. On other hosts, ask support for the requests from Google’s crawlers and the status code each one got.
7. Fix it, then ask Google to look again. On the issue, select View fix, then Request website check. Google says Google may take up to 12 hours to review your website. and If you fix these issues, the errors should disappear within a couple of days.

A note on Cloudflare: Cloudflare says You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules. So an “allow Google” rule won’t stop Bot Fight Mode from challenging a request. If it’s the cause, Cloudflare’s own advice is to turn it off, or to move to Super Bot Fight Mode, which supports skip rules.

On Shopify, WooCommerce and other platforms

Shopify

• Password page. Shopify’s Google & YouTube channel requirements say your store can’t be in private mode. Check Online Store → Preferences and turn the password off once you sell.
• robots.txt. Shopify’s default file works for most stores. A theme can change it with a robots.txt.liquid template, and Shopify calls that an unsupported customization that can cost you traffic (Shopify Help: Editing robots.txt). If your theme has one, read it for Google crawler groups.
• Your own Cloudflare in front of Shopify. Cloudflare calls this setup O2O. With it, your own Cloudflare zone’s bot settings, rate limits and Workers apply to Google’s requests before Shopify sees them. That’s where to look first.
• Country redirect and blocking apps. An app that redirects or blocks visitors by country can catch Google’s crawlers, which often come from the US. Make sure US visitors reach the product page your feed links to.

WooCommerce

• Coming soon mode. WooCommerce says When Coming soon mode is enabled, your site is in a development state and not accessible to the public. New stores start in it. Check WooCommerce → Settings → Site visibility and set it to Live (WooCommerce: Coming soon mode).
• Security plugins. Firewall, rate-limit and country-blocking settings in WordPress security plugins can block Google’s crawlers. Open the plugin’s live traffic or blocked-requests log and search for Google. Allow Google’s real crawlers, not anything that only claims the name.
• Your host’s own layer. Many hosts run their own firewall or CDN in front of WordPress, and it doesn’t show in your settings. If your own settings look clean, ask the host what Google’s crawlers received.
• Slow pages. An uncached product page can time out under a burst of crawling. Caching helps.

Other platforms

On Wix, Squarespace, BigCommerce and others, check for a site password, an “under construction” page or member-only pages, then any CDN or firewall you added. On a custom site, check rate limits, firewall rules and country blocks, and read your access logs for Google’s requests.

What we see in real cases

Paraphrased from merchants’ posts in the Shopify, Cloudflare and Google communities, and public reports, before this post:

• A WooCommerce store kept getting “Product page unavailable” with a client 4xx error. The owner turned Bot Fight Mode off, added a WAF skip rule and even paused Cloudflare, and Google still got errors. They suspected a Cloudflare layer run by their host. The thread closed with no replies.
• Two Shopify stores with their own Cloudflare zone in front of Shopify found Google’s crawlers getting HTTP 429 answers. One logged hundreds of 429s to Googlebot in under a day, with a Cloudflare Worker in the path.
• In a long Shopify thread, “Product page unavailable” cleared each time a merchant requested a website check, then crept back over the following weeks. One store went from none to hundreds of affected products in a single day in early June.
• In the same week, an ads specialist publicly reported “Product page unavailable” across more than ten accounts they manage, overnight, on pages that were live. Google hadn’t explained it when this post was written. If your pages pass every test above, request a website check and keep a record of what you tested.
• One merchant reviewing a Misrepresentation suspension found that robots.txt was blocking their policy pages. A robots.txt block can matter beyond product listings.

Our guide to the landing page not working issue has the short version of these steps. If products are missing from Shopping without any issue in Merchant Center, our diagnostic guide to products not showing covers the other causes. Our free store check opens your homepage and product pages from outside and reports a password page, and says when a bot check stopped it from reading a page. It can’t see what Google’s crawlers get, because a firewall can treat Google differently from any other visitor.

FAQ

Why does Google say my product page is unavailable when it loads for me?

Because Google’s crawler isn’t you. It arrives from Google’s own addresses, often in the US, with its own name, and a firewall, bot setting, rate limit, country block or password page can treat it differently from your browser. Google’s fix page asks for a landing page that is accessible from any device, location, or browser.

Which Google crawlers need access to my product pages?

When this post was written, Google’s Merchant Center page named AdsBot-Google for ads and Googlebot for organic search. Google’s crawler documentation adds Storebot-Google for Google Shopping surfaces. In our reading, the safe setup is not to block any of the three in robots.txt, your firewall or your host’s settings.

Can Cloudflare block Google Merchant Center from my product pages?

It can, in our reading of merchants’ reports: a bot challenge or rate-limiting rule that a person never sees can answer Google’s crawler with an error. Check Cloudflare’s security events for requests from Google that were challenged or blocked, and note that Cloudflare says Bot Fight Mode can’t be skipped with WAF custom rules.

How long does “Product page unavailable” take to clear after a fix?

Google says a website check you request may take up to 12 hours, and you can’t send another request during that time. It also says “If you fix these issues, the errors should disappear within a couple of days.”