Download the Internet Index for free!
Sunday, March 8th, 2009Well, kind of… dotnetdotcom.org aka Dotbot
You might have noticed a new little critter gnawing at your web server:
Mozilla/5.0 compatible; DotBot/1.1; http://www.dotnetdotcom.org/crawler@dotnetdotcom.org)
The few Seattle based guys (pseudonym on their web site) promise an index of the web available to everybody. They intent to release as much information about the web’s structure (linking) and content as possible. For a fee to cover their costs, though.