July 27, 2019

Robot.txt File

If you have a website, you might be familiar with robot.txt file. This file that search engines use to determine what directories should get indexed.

Google has a robot.txt checker tool, which is pretty good. However, the folks over at websiteplanet.com created a useful robot.txt audit tool. Now you can see if your robot.txt file is up to specs! Simply enter the location of your robot.txt file and then go!

Website Description

It's never been easier to test the accuracy of your robots.txt file. Just paste your complete URL, with /robots.txt, click enter, and your report will be ready quickly.

Taking into account both the Robots Exclusion Standard and spider-specific extensions, our robots.txt checker will generate an easy to read a report that will help correct any errors you may have in your robots.txt file.

Robot Txt
https://www.websiteplanet.com/webtools/robots-txt/

Robot.txt Tips

Three tips on using the Robot.txt file:

Put in a fake directory that only exists in the robot.txt file to indicate that it shouldn't be searched. Then check your logs to see if anyone has hit it. This is a good way to see if someone is harvesting the "forbidden search directories."

Yoast.com created The ultimate guide to robots.txt. This is useful to understand all the latest options that robot.txt files have.

Doesn't hurt to check out other websites on how they have their robot.txt file setup.

Robot.txt References

Useful Robot.txt Guides:

Comments

Add your Comments

Feel free to leave a comment about this post.

- Feel Free to add HTML to your comment!