The public /robots.txt response can come from more than WordPress. A physical file, SEO plugin, WordPress virtual output, CDN rule or deployment template can all become authoritative. Fixing the visible line without finding its source is likely to be temporary.
Fetch the live file from the canonical host
Save the body and response headers, then test the affected URL against each applicable User-agent group. A broad path rule can block more than the author intended.
Find which layer owns the directive
- physical
robots.txtin the document root; - WordPress search-visibility setting;
- SEO/security plugin filters;
- CDN or edge configuration;
- staging/deployment templates copied to production.
Change only the unintended rule
Private, duplicate or utility paths may still need crawl controls. Do not replace a carefully scoped file with an unrestricted generic version.
Validate the production response after cache purge
Request the exact canonical host again and confirm that the change survives deployment and edge caching. Search Console live inspection can then verify representative URLs.
Connect crawl blocking to what Google is actually seeing
If unexpected pages are already appearing in search, compare the SEO-spam visibility diagnostic. If migration left conflicting domains or canonicals, review the old-domain migration checks.
Do not use robots.txt as access control
Robots directives are crawl instructions, not protection for sensitive data. Authentication and authorization must enforce privacy.
Verification
Googlebot should be allowed to crawl intended public URLs, deliberately restricted paths should remain controlled, and the live rule should persist correctly after cache refresh and deployment.
WP REPAIR INCIDENT STANDARD
Reconstruct the incident before fixing it
A crawler warning can also come from a rule aimed at an overly broad path or user agent.
- 1
Fetch the live robots.txt from the canonical host and record headers.
- 2
Test the affected URL against every applicable User-agent group.
- 3
Check for a physical file before WordPress and plugin filters after it.
- 4
Inspect CDN, staging-protection and deployment templates.
What must be verified
- Googlebot is allowed to crawl intended public URLs.
- Private/admin paths remain appropriately protected.
- The rule persists correctly through deployment and cache refresh.
Official technical sources
This guide explains the diagnosis. If the site is affected now, the intervention should preserve a rollback path and verify the real business journey.
See the emergency repair service →ABOUT THIS SYMPTOM
Frequently asked questions about this guide.
If I fix the Discourage search engines setting in WordPress, is that always enough to unblock Googlebot?+
Not always. A physical robots.txt file uploaded to the server overrides WordPress's virtual output, and a CDN can also keep serving a cached or environment-specific version, so the setting change needs to be verified against the live response, not assumed to be the only source.
Should robots.txt be used to hide sensitive or private content from being found?+
No. robots.txt is a crawling instruction, not an access control — sensitive data should be protected with authentication or authorization, since a Disallow rule does not prevent the URL from being accessed directly.
Can a security plugin block crawling without anyone touching the visibility setting?+
Yes. A plugin or security rule can independently emit a broad Disallow directive that blocks crawlers, entirely separate from the core WordPress search-visibility checkbox, so both sources need to be checked.
After fixing the rule, is it enough to just look at the live robots.txt file to confirm the fix worked?+
You should also purge edge caches and test representative URLs, since a CDN can keep serving the old file even after the authoritative source is corrected, and Search Console's live inspection tool helps confirm Google is actually seeing the updated rule.