robots.txt & sitemaps

robots.txt and XML sitemaps get mentioned in the same breath, but they do opposite jobs. robots.txt tells crawlers where they may not go. A sitemap tells them where you would like them to go. Confusing the two is behind most of the indexing problems we untangle for Florida e-commerce brands.

robots.txt controls crawling, not indexing

This is the distinction that trips up almost everyone. A URL you block in robots.txt can still appear in Google results — usually with no description — if other pages link to it, because Google never fetched the page to read your instruction. If your goal is to keep a page out of the index, robots.txt is the wrong tool.

The file lives at the root of your domain (yourstore.com/robots.txt) and is read before Google crawls anything else. For a typical store, the safe pattern is: allow everything by default, point to your sitemap, and disallow only the URLs that generate endless low-value crawling.

• Disallow internal search results (/search, ?q=) — they spawn infinite thin URLs
• Disallow cart, checkout and account pages — no SEO value, sometimes sensitive
• Watch faceted-navigation parameters (?color=, ?size=, ?sort=) that multiply URLs
• Never disallow your CSS or JS — Google needs them to render and judge the page
• Always include the full absolute URL to your sitemap

One line can do real damage. A stray Disallow: / pushed live during a redesign will, over a few days, strip an entire site from search. Whenever you touch this file, test it in Search Console before and after.

Disallow vs noindex: the decision to get right

Use noindex (a meta robots tag or HTTP header on the page) when you want a page crawled but kept out of results — thin tag pages, filtered listings, or expired promotions you still want reachable. Use Disallow in robots.txt when you want crawlers to stop wasting time on URLs that have no business being fetched at all, like faceted parameter explosions.

The trap: you cannot do both to the same URL. If you Disallow it, Google can't crawl it, so it never sees the noindex tag — and the page can linger in the index indefinitely. To reliably remove a page, allow the crawl and serve noindex until it drops out, then block it later if you wish.

XML sitemaps that actually help

A sitemap is a curated list of the URLs you consider canonical and want indexed. For a store it is a quality signal, not a dumping ground. The most common mistake we see is a sitemap stuffed with URLs that redirect, 404, are noindexed, or are non-canonical — which teaches Google to trust the whole file less.

• Include only indexable, canonical, 200-status URLs
• Exclude redirects, error pages, noindexed pages and parameter duplicates
• Keep each file under 50,000 URLs and 50MB; split large catalogs with a sitemap index
• Let your platform regenerate it automatically as products come and go
• Keep lastmod dates honest — faked freshness helps nothing

Shopify, WooCommerce and PrestaShop all generate sitemaps for you. The real work is auditing what they produce: out-of-stock or unpublished products, orphaned collection pages, and staging URLs all tend to sneak in.

Submit it, then monitor the gap

Reference the sitemap in robots.txt and submit it in Google Search Console under Sitemaps. Then watch the report — Google shows how many URLs it discovered versus how many it indexed. A large gap is a signal worth investigating, not a number to ignore. For the mechanics of that report and the URL Inspection tool, see our Google Search Console guide, and Google's own robots.txt documentation for exact syntax.

Why this is really about crawl budget

On a large catalog, every crawler request spent on a filtered duplicate is a request not spent on a product you want ranked. Tightening robots.txt and cleaning your sitemap is how you steer that attention where it earns money. We go deeper in the crawl budget guide and the broader technical SEO guide. If you would rather have it handled end to end, our technical SEO service treats crawl control, indexing and sitemaps as one workstream.