Robots.txt Generator tool helps you generate the robots.txt file.
You can control which files crawlers may access on your site with a robots.txt file. A robots.txt file lives at the root of your site. So, for site www.example.com, the robots.txt file lives at www.example.com/robots.txt. robots.txt is a plain text file that follows the Robots Exclusion Standard. A robots.txt file consists of one or more rules. Each rule blocks or allows access for a given crawler to a specified file path in that website. Unless you specify otherwise in your robots.txt file, all files are implicitly allowed for crawling. Hence it's important to generate this file and our small seo tool will come in handy for this purpose!