Free robots.txt & sitemap checker
Check whether a site's robots.txt is blocking search engines and whether its sitemap is present and valid — in one click.
The mistake that deindexes sites
The most common — and most damaging — SEO regression is a Disallow: / shipped from a staging environment to production. It quietly tells Google to stop indexing the entire site, and traffic falls off a cliff weeks later. A missing or unreferenced sitemap slows discovery too. Checking these takes seconds; monitoring them on client sites turns a silent catastrophe into an instant alert.
Frequently asked questions
What does this check?
Whether robots.txt exists, whether it accidentally blocks all crawlers (Disallow: /), whether sitemap.xml exists and is valid, and whether the sitemap is referenced from robots.txt.
What's the most dangerous robots.txt mistake?
A leftover 'Disallow: /' from a staging site shipping to production — it tells Google not to index anything and can deindex the whole site. This catches it instantly.
Do I need a sitemap?
It's not mandatory, but a sitemap helps search engines discover and prioritize your pages — especially for larger or newer sites.
Sitefolio monitors robots, sitemaps and on-page SEO across your portfolio and alerts you on any change.
Get started free