The past two weeks have seen an upswing in posts around here, and the consequence is that more spiders stop by, and the consequence is that they find the search links on the right. When the spiders hit the search links in quick succession without loading anything else, it digs a big crater where the server used to be. Well, I'm tired of that.
There are a ton of nasty bots that aren't obeying the robots.txt, and really I think it can't be relied upon to prevent the most heinous bots from destroying a site's productive page serving. I've thrown some Apache modules at the issue, but they don't seem to help. I've specifically excluded sections of the site from certain user agents, and that seems to do well, but there's no avoiding getting trounced by these freaking ill-behaved spiders and comment spamming bots.
So I wrote a new Habari plugin that checks the server load and sends a 503 (Service Unavailable) when the load gets too high. A custom template is employed for the 503, so even though the site doesn't provide content, it still looks like it's trying. The settings are configurable, so I can turn certain taxing features of the site off temporarily while there's moderate load, and send the whole site into standby if the load hits the roof. When the load returns to normal, the walls automatically drop....
more