Control how search engines crawl your site. Block sensitive directories and AI training bots to protect your content.
A robots.txt file is one of the most powerful tools in a webmaster's technical SEO arsenal. It acts as a set of instructions for search engine crawlers, telling them which parts of your site should be explored and which should be ignored.
Google only spends a limited amount of time on each site. By blocking low-value pages, you ensure Googlebot focuses on your most important content.
Blocking AI scrapers (like GPTBot) prevents your original data and content from being used to train third-party LLMs without your permission.
Hiding login pages and system directories (like /wp-admin/) from search results reduces the visibility of sensitive entry points to hackers.
Once you've generated and downloaded your file, upload it to the very root of your domain (e.g., yoursite.com/robots.txt). Search engines will look for this file automatically every time they visit your site.