Question 1

Does robots.txt affect all search engines or just Google?

Accepted Answer

Robots.txt applies to any crawler that respects the Robots Exclusion Protocol — which includes Google, Bing, and most legitimate web crawlers. The User-agent: * directive applies to all crawlers; you can also write rules for specific crawlers using their user-agent names (e.g., User-agent: Googlebot for Google only). Malicious bots and scrapers don't respect robots.txt, so it provides no security benefit — it's purely a crawl directive for legitimate bots. AI training crawlers vary in their compliance; some respect robots.txt, others (particularly older or less established crawlers) may not.

Question 2

How quickly does Google respond to robots.txt changes?

Accepted Answer

Google re-fetches robots.txt regularly — typically every few days for active sites. Once a blocking rule is removed, Google should resume crawling the previously blocked pages within a few crawl cycles. If you've accidentally blocked important pages and need faster recovery, use the URL Inspection tool to request indexing for the most critical pages after removing the block — this queues them for crawling sooner than waiting for the regular schedule. Recovery from a complete site block (Disallow: /) can take one to four weeks to fully restore indexation, depending on your site's crawl frequency.

Question 3

What should a clean, minimal robots.txt look like?

Accepted Answer

For most WordPress sites, a clean robots.txt is minimal: allow everything except specific admin and internal paths that should never be indexed or crawled. A typical clean version: User-agent: * with Disallow: /wp-admin/ (the admin panel) and Disallow: /wp-includes/ (core WordPress files), plus a Sitemap line. Everything else should be allowed. Avoid blocking /wp-content/ (which contains your theme and plugin CSS/JS), search result pages unless they're causing significant crawl waste, and any path that leads to canonical, indexable content you want ranked.

Question 4

Can robots.txt be used to block AI training crawlers?

Accepted Answer

You can block specific AI training crawlers by name in robots.txt — for example, User-agent: GPTBot with Disallow: / blocks OpenAI's training crawler. Most major AI companies (OpenAI, Anthropic, Google) publish their crawler user-agent names and respect robots.txt rules for training data collection. However, crawlers used for AI Overviews and AI search results are different from training crawlers — blocking Google's AI training crawler doesn't block Googlebot and won't prevent your content from appearing in AI Overviews. The distinction matters: if you want to avoid AI training but still rank in search, target the specific training crawler user-agents rather than Googlebot.

Question 5

Why would a blocked page still show up in Google search results?

Accepted Answer

Because robots.txt blocks crawling, not indexing. If other sites link to a blocked URL, Google can still add it to the index based on those external signals — it just can't read the page's actual content, so the result shows a generic message like "No information is available for this page" instead of a normal title and description. This is usually worse for click-through rate than not being indexed at all, since the snippet looks broken to searchers. If you genuinely don't want a page indexed, unblock it in robots.txt and add a noindex tag instead.

Question 6

Should robots.txt checks be part of a standard launch checklist?

Accepted Answer

Yes — given how cheap the check is and how expensive the mistake is, robots.txt verification belongs on every migration, replatform, and major deployment checklist, not just the SEO team's radar. A simple step works: visit yourdomain.com/robots.txt immediately after any production deploy and confirm there's no blanket Disallow rule and no unexpectedly broad path matches. This takes under a minute and catches the most damaging class of technical SEO mistake before it has days to compound.

Technical SEO

Robots.txt Mistakes That Quietly Deindex Your Site

Redirect Chains and Loops: A Quick Audit and Fix

Page Speed for WordPress: A Realistic Optimisation Stack

Structured Data That Actually Earns Rich Results

Mobile-First Indexing: What Still Trips Sites Up

International SEO Basics: Hreflang Without the Headaches