For every webmaster a googlebot visit to his website is a small event. So every webmaster will try to be as liberal regarding what he allows crawlers as googlebot, msnbot and yahoo!slurp as possible.
Each of this robots will identify using a special User Agent string which is known by everyone. So once they have the User Agent 'put on' we automatically think they are the real deal. But is the User-Agent Real or Fake?
But believe or not many out there browse sites claiming to be googlebot in order to gain advantages over competition and pass undetected. This is both a security hazard as well as a bandwidth one.
See below some real life examples when validation is mandatory and how lack of validation can cause serious problems. Googlebot validation saves lives. MSNbot and Yahoo! Slurp validation also.
Let's say you offer a membership site. You have quality content that needs to be found in order to attract costumers but you will only share it for a fee.
In order to have clients you need search engine visitors so you need the sensitive content indexed. So you put nocache in robots meta tag and you just feed the page to the visitors who match the user agent profile.
But the user-agent is by far most spoofed attribute in the HTTP protocol. So anyone with basic knowledge can use PHP and cUrl to retrieve your pages claiming to be a well known robot.
This way your membership sensitive content will be available to anyone claiming to be a webrobot you trust.
Some web hosting services are not really that generous with bandwidth. And on the other hand you have several hungry robots hitting you as hard as you can take it and putting great pressure on your rusty webserver.
I bet you would love to find the fake ones and send them on the other side where they belong and this way saving you valuable bandwidth.
I do not believe in such thing. When someone claims to be someone he's not, he's not usually doing it to help you or for your benefit. Any web crawler claiming to be an established and trusted one is 100% up to no good!
Even when a coder tests his site using fake user agents it means he has something to hide and tests his site for cloaked content.
Only way to answer this is: I don't know! You must verify it to issue a definitive opinion on the validity of the identification data your visitor is using.
All major search engines have a simple feature. To verify Googlebot, msnbot, Slurp or Ask follow the next 2 easy steps. The procedure is easy and goes like this:
Plain and simple. So what should you expect to find in the hostname?
If the host passes the above verification it means it is a trustable webrobot. If a webrobot contains a fake user agent it will not pass the test above so block it.
I create a simple Bot Verification PHP Script By DNS Lookup. Obviously data is cached if a mysql connection is available this way avoiding slowdowns. This can also be used for a really dirty trick : link exchange cloaker!
Keep in mind nothing is 100% guaranteed so if you have not really legit propouses of using this you might / will go down!
Post Feedback