What is robots.txt?
robots.txt is a small text file at the root of your site that tells search engine crawlers which parts of the site they may visit.
What it does#
Before crawling your site, well-behaved search engines fetch yoursite.com/robots.txt and follow its rules. It is a set of requests, not a lock: it keeps crawlers out of pages that waste their time (internal search results, cart pages), but it is not a security measure and should never be used to hide private content.
The common rules#
A typical file is only a few lines:
User-agent: *: which crawler the rules apply to (*means all)Disallow: /admin/: paths crawlers should skipAllow: /: paths crawlers may visitSitemap: https://yoursite.com/sitemap.xml: where the sitemap lives
The mistake to watch for is Disallow: / left over from a staging environment. That single line asks search engines to skip your entire site, and it is a depressingly common reason a freshly launched site never appears in Google.
How Weblooks helps#
The SEO scan fetches your site's robots.txt, checks it exists, and flags a file that blocks everything. It also checks each page for a noindexmeta tag, the page-level cousin of robots.txt, so a leftover "hide this from Google" flag is caught before launch instead of three months after.