I decided to revive some older posts which might have gone un-noticed at that time due to very few readers. Quite funny is I got two forum links on this one from same forum and two different memebers: elhoim and Mark (the admin probably).
This post is worth reading and paying attention to!
Google, MSN and Yahoo! provided us with a way to verify crawlers which can be very legit used in today's world wild web to protect yourself from proxy hijacking! I actually have about 5 IP addresses that hit me with fake user agents and got the diserved HTTP 500 Error Code in the past 48 hours!
Google went one step further and provided the X-Robots-Tag to allow use to set the noindex, nofollow, noarchive rules on … other file formats for example. Very usefull new trick indeed. For a HTML site .htaccess can be combined with PHP to alter Robots tag based on URL and no changes would be needed to HTML files. All the magic goes on in the HTTP Headers.
This two features are awsome and Google did a great job.
The Googlers are smart and I'm curious if they ever thought how this can hit link exchange. Many link exchange scammers out there can use this very easily not to pass links anymore! Pages are cached and nofollow can be easily found. Uncached pages will raise a question mark instantly …
… but headers are not recorded anywhere. So passing a nofollow in header on to Googlebot will kill all the link juice trying to leave your link exchange pages. And, even on human review, the page would pass undetected unless that would be what the reviewer is looking for.
First you have the bot verification code and you feed nofollow in the HTTP Header to the Robots. You remove the Robots META Tag from HTML (to look legit) and … Voila! One way links for the take. The ways to verify legit link exchangers will fail to detect this.
I do not think MSN and Yahoo! are playing by this rule … YET!
Then again, it might just be me and my dirty mind!
