SEO
Sitemap or robots.txt: they do different jobs
One advertises pages. The other restricts crawlers. Confusing them is how sites accidentally block the file that was supposed to help, or publish a list of URLs nobody is allowed to fetch.
Check a sitemapThese two files sit next to each other at the root of every site, look equally technical, and do opposite things. A sitemap is a suggestion: here are the pages, please look. Robots.txt is a restriction: keep out of these paths. Neither guarantees anything on its own.
robots.txt: one job, badly supported
It is a plain text file at /robots.txtwith groups of rules per user agent. Disallow stops a crawler fetching a path. That is the whole mechanism.
- It does not stop a page being indexed if other pages link to it — a URL can appear in results with no snippet, because the crawler was told not to fetch it but learned it exists.
- It does not remove a page. For that you need
noindex, which requires the crawler to be allowed to fetch the page. - It is advisory. Well-behaved crawlers obey; scrapers do not.
The classic self-inflicted wound: Disallow: /during a staging deploy, forgotten on production, while the sitemap cheerfully advertises the whole site. The sitemap is fine; nobody is reading it.
sitemap.xml: an advertisement
XML, one <url> per page with an optional lastmod. The protocol caps a single file at 50,000 URLs and 50 MB uncompressed; beyond that you need a sitemap index pointing at children. Listing a URL is a hint that it exists, not a promise it will be crawled, and certainly not a promise it will rank.
Two details that decide whether it helps at all:
lastmodmust mean something. If every URL carries today's date because it is stamped at build time, the field is noise and gets ignored. Real modification dates only.- The sitemap must list fetchable URLs. If it lists paths that robots.txt disallows, or redirects, or 404s, the file teaches the crawler that it is unreliable.
How they should cooperate
- robots.txt allows everything you want indexed, and names the sitemap with a
Sitemap:line. - The sitemap lists only canonical, 200-status URLs that are not disallowed.
- Pages you do not want indexed get
noindexand stay crawlable — do not disallow them, or the noindex is never seen.
Checking both in two minutes
- Fetch
/robots.txt. Is there aDisallow: /you did not intend? Does the sitemap line point at a file that returns 200? - Fetch the sitemap. Does it open with
<sitemapindex>? If so, open one child — that is where the URLs actually are. - Pick five URLs from it and check each returns 200 without a redirect.
- Count them. A sitemap with 12 URLs for a site with 400 pages is describing a fraction of the site.
Sitemap checker fetches and parses the file, follows an index into its children, and reports the URL count, the status of a sample, and whether anything in it is blocked by the site's own robots.txt. There is also a robots.txt checker for the other half.
The sentence to remember
robots.txt is a fence, a sitemap is a signpost. Putting the signpost behind the fence is the mistake that costs months.