for making me waste two hours from my Sunday fighting your crawler bot. It's now blocked although about 30% of attempts on the server are still from it (down from 55%, and weekends are busy for this site). I understand, dear 80legs, that you have a service to sell to your clients, but unfortunately we need our site to be up... :)
Apologies to Bad Robot for the terrible use of their logo :)
Google+: View post on Google+
Hey :)
ReplyDeleteFirst things: Your Comment-Login over twitter is not working - It gives an PHP Exception with OAuth...
Second things:
I hate 80legs.com - The Bad Service is a DDoS-Service for many Webservers.
My solution is: Block 80legs not over the Server-Layer (e.g. Webserver filter/htaccess/Vhost with matching the 80.legs Useragent), block the f**** Service over the Routing/Backbone/Firewall Layer of the Hosting-Infrastructure if possible.
Problem: normal Users (Webspace cutomers) dont have permissions to do this. All Hosting-Providers must make that steps. We have thousend of servers and we have block the Service of 80legs successfully before the Requests comes to the rootservers and to their webservers...
If you are an Hosting-Provider: Block the shit service you customers make it happy!
Thank you for the heads up on the Twitter integration, it's off until I get time to fix it...
ReplyDeleteAs for 80legs, I blocked them via robots.txt, apache level and on application level... :)