QA Graphic

Robot.txt File

Audit your robot.txt file to make sure it is up to code.

If you have a website, you might be familiar with robot.txt file. This file that search engines use to determine what directories should get indexed.

Google has a robot.txt checker tool, which is pretty good. However, the folks over at websiteplanet.com created a useful robot.txt audit tool. Now you can see if your robot.txt file is up to specs! Simply enter the location of your robot.txt file and then go!

Website Description

It's never been easier to test the accuracy of your robots.txt file. Just paste your complete URL, with /robots.txt, click enter, and your report will be ready quickly.

Taking into account both the Robots Exclusion Standard and spider-specific extensions, our robots.txt checker will generate an easy to read a report that will help correct any errors you may have in your robots.txt file.

Robot Txt
https://www.websiteplanet.com/webtools/robots-txt/

Robot.txt Tips

Three tips on using the Robot.txt file:

Put in a fake directory that only exists in the robot.txt file to indicate that it shouldn't be searched. Then check your logs to see if anyone has hit it. This is a good way to see if someone is harvesting the "forbidden search directories."

Yoast.com created The ultimate guide to robots.txt. This is useful to understand all the latest options that robot.txt files have.

Doesn't hurt to check out other websites on how they have their robot.txt file setup.

Robot.txt References

Useful Robot.txt Guides:

 

Comments

Add Comments

Name:
Comment:

 

About

I will highlight a fun Internet site every Saturday. The basic premise is to highlight sites that I regularly visit for a variety of reasons. All site recommendations are mine and are unsponsored. By highlighting these websites, I hope to provide value to those who may not know of the sites or who might be interested in visiting the sites. I also want to share the sites that I find valuable in the hopes that others will find them just as beneficial.

Schedule

WednesdaySnagIt for QA
ThursdayPython
FridayMacintosh
SaturdayInternet Tools
SundayOpen Topic
Monday Media Monday
TuesdayQA