Robot.txt File
Audit your robot.txt file to make sure it is up to code.
If you have a website, you might be familiar with robot.txt file. This file that search engines use to determine what directories should get indexed.
Google has a robot.txt checker tool, which is pretty good. However, the folks over at websiteplanet.com created a useful robot.txt audit tool. Now you can see if your robot.txt file is up to specs! Simply enter the location of your robot.txt file and then go!
Website Description
It's never been easier to test the accuracy of your robots.txt file. Just paste your complete URL, with /robots.txt, click enter, and your report will be ready quickly.
Taking into account both the Robots Exclusion Standard and spider-specific extensions, our robots.txt checker will generate an easy to read a report that will help correct any errors you may have in your robots.txt file.
https://www.websiteplanet.com/webtools/robots-txt/
Robot.txt Tips
Three tips on using the Robot.txt file:
Put in a fake directory that only exists in the robot.txt file to indicate that it shouldn't be searched. Then check your logs to see if anyone has hit it. This is a good way to see if someone is harvesting the "forbidden search directories."
Yoast.com created The ultimate guide to robots.txt. This is useful to understand all the latest options that robot.txt files have.
Doesn't hurt to check out other websites on how they have their robot.txt file setup.
Robot.txt References
Useful Robot.txt Guides:
- Create a robots.txt file - Search Console Help
- How to Create a Robots.txt File - Bing Webmaster Tools
- Blocking the Baiduspider