Robots.txt Guide for Small Websites
Understand what robots.txt can control, what it cannot do, and how to create a simple crawl policy safely.
This guide focuses on the practical decisions that matter when completing this task. Use it with the related GetTools utility, and always review the output before using it in an important workflow.
What robots.txt controls
Robots.txt gives crawler directives for paths on a host. It is mainly a crawl-management file, not a security system.
Robots.txt is not a reliable noindex method
Blocking a URL from crawling does not necessarily prevent the URL from appearing in search results. For pages that must not be indexed, use an appropriate indexing directive while allowing the crawler to see it.
Keep rules simple
Start with the minimum rules you need. Broad disallow patterns can accidentally block important pages, CSS, JavaScript or images.
Use the correct host location
The file belongs at the root of the relevant host, such as /robots.txt. Rules apply to that host and protocol context, so verify the production version after deployment.
Sitemaps can be referenced
A Sitemap line can point crawlers to your XML sitemap. Use the final canonical sitemap URL.
Test before and after deployment
Review generated rules carefully, upload the file, request it directly in a browser, and monitor crawl/indexing reports for unexpected changes.
Quick checklist
- Confirm the destination requirements before you start.
- Keep an original copy when changing files or established URLs.
- Review the generated result rather than assuming automation is perfect.
- Use the related GetTools utility for the mechanical step, then make the final human check.