Skip to main content
BrowserStay

Command Palette

Search for a command to run...

SEO

Sitemap or robots.txt: they do different jobs

One advertises pages. The other restricts crawlers. Confusing them is how sites accidentally block the file that was supposed to help, or publish a list of URLs nobody is allowed to fetch.

Check a sitemap

These two files sit next to each other at the root of every site, look equally technical, and do opposite things. A sitemap is a suggestion: here are the pages, please look. Robots.txt is a restriction: keep out of these paths. Neither guarantees anything on its own.

robots.txt: one job, badly supported

It is a plain text file at /robots.txtwith groups of rules per user agent. Disallow stops a crawler fetching a path. That is the whole mechanism.

  • It does not stop a page being indexed if other pages link to it — a URL can appear in results with no snippet, because the crawler was told not to fetch it but learned it exists.
  • It does not remove a page. For that you need noindex, which requires the crawler to be allowed to fetch the page.
  • It is advisory. Well-behaved crawlers obey; scrapers do not.

The classic self-inflicted wound: Disallow: /during a staging deploy, forgotten on production, while the sitemap cheerfully advertises the whole site. The sitemap is fine; nobody is reading it.

sitemap.xml: an advertisement

XML, one <url> per page with an optional lastmod. The protocol caps a single file at 50,000 URLs and 50 MB uncompressed; beyond that you need a sitemap index pointing at children. Listing a URL is a hint that it exists, not a promise it will be crawled, and certainly not a promise it will rank.

Two details that decide whether it helps at all:

  • lastmod must mean something. If every URL carries today's date because it is stamped at build time, the field is noise and gets ignored. Real modification dates only.
  • The sitemap must list fetchable URLs. If it lists paths that robots.txt disallows, or redirects, or 404s, the file teaches the crawler that it is unreliable.

How they should cooperate

  1. robots.txt allows everything you want indexed, and names the sitemap with a Sitemap: line.
  2. The sitemap lists only canonical, 200-status URLs that are not disallowed.
  3. Pages you do not want indexed get noindexand stay crawlable — do not disallow them, or the noindex is never seen.

Checking both in two minutes

  • Fetch /robots.txt. Is there a Disallow: / you did not intend? Does the sitemap line point at a file that returns 200?
  • Fetch the sitemap. Does it open with <sitemapindex>? If so, open one child — that is where the URLs actually are.
  • Pick five URLs from it and check each returns 200 without a redirect.
  • Count them. A sitemap with 12 URLs for a site with 400 pages is describing a fraction of the site.

Sitemap checker fetches and parses the file, follows an index into its children, and reports the URL count, the status of a sample, and whether anything in it is blocked by the site's own robots.txt. There is also a robots.txt checker for the other half.

The sentence to remember

robots.txt is a fence, a sitemap is a signpost. Putting the signpost behind the fence is the mistake that costs months.