For years now I have been aghast at the terribly impolite behavior of some bots who spider my site. It is extremely common to see many successive requests to the same URL, often a dozen or so within a second from the same IP address. All these requests succeed so why does the bot repeat them?
I chalked it up to the bot creators being idiots but a couple of weeks ago I sat down to write a spider for a private project and ran into some of the same issues. My spider doesn’t make repeated requests for a single host because I am sensible but I now I see why these bots behave this way.
It is not as easy as I thought and perhaps I owe the bot creators an apology.
Here is how writing spider starts.
One important step that I missed out here is that the processor must check the database for URLs that you have already spidered. We don’t want to add the same URL to the queue 50 times just because 50 other pages refer to it. Remember we are trying to be nice here. We also want to avoid endless loops in the quite common situation where two pages link to each other.
But assuming we have dealt with that we will run into a second problem. This design is slow. Really slow. The issue is that each request can take seconds during which our spider is effectively doing nothing but waiting.
This all sounds great and works fine - except that now your spider is no longer polite. Depending on how your database works, it is possible (and actually very likely) that multiple copies of the same URL will be added to the queue from multiple processors. Such a design can really hammer a server without trying too hard. This can be solved by carefully locking your datastructures while checking for existing records but this slows things down.
This is the stage that I suspect these impolite bots have reached. They have some multi-process architecture that doesn’t properly check for existing records in a safe way. Because requests are cheap and performing needless requests only slows them down slightly, the operators do not care that they bots slam sites. They don’t pay for the chaos they cause.
Remember when I said I might owe those bot creators an apology. They are not getting one, because fuck them.
Here are some of the considerations that these bots developers should be taking into account:
Just because you have twenty thousand different URLs from somebodyelses.server.com doesn’t mean that you should hit that server with your 128 processor bot. Some mechanism needs to be in place to rate limit request to a single server. Even limiting the rate to once a second per host would go a long way to making these bots nicer.
Other issues I came across while writing my own bot were quite subtle. You really need to normalize your URLs as encounter them. The following hostnames are all equivalent so simple string matching doesn't suffice:
| Hostname | Notes |
|---|---|
| example.com | If only it was always so simple. |
| Example.com | Hostname are not supposed to have capital letters but you will find plenty in the wild. |
| example.com. | The rarely seen but valid trailing dot. |
This doesn’t include the dozens of ways to encode the path, or domains resolving to the same address, or punycode. It is a mess.
Finally, it is pretty common to have multiple hostnames redirecting to a single server via 301 Moved Permanently or similar mechanism. It would be impolite to spider a site 3 times just because it had 3 names.
Despite these difficulties, some of which have no real solution and must be dealt using heuristics, it is important to ensure that any bot you deploy is nice enough not to affect the sites you spider. Anything less is not just rude but causes actual pain for site owners.