I Am Experimenting with Blocking HTTP1.1

Most of the traffic I get on this site is bots - it isn't even close. And, for whatever reason, almost all of the bots are using HTTP1.1 while virtually all human traffic is using later protocols.

I have decided to block v1.1 traffic on an experimental basis. This is a heavy-handed measure and I will probably modify my approach as I see the results.

Already I have noticed that mastodon page previews are fetched using HTTP1.1, so some sort of agent filtering might need to be added. I already make exceptions for feeds (rss.xml, atom.xml) that are designed to be read by bots anyway.

You may think that blocking bots is self-defeating, how are search engines going to spider my site? I have decided I don't care - I get so little organic traffic from search engines these days that I do not think it matters.

Implementation

Most other people who are trying this (see this informative post, for example) are using nginx but I am using caddy to serve sheep.horse. It seems like it is impossible to completely disable HTTP1.1 with caddy (at least for the moment) but I did the next best thing in the Caddyfile:

# Return an error for clients using http1.1 or below - these are assumed to be bots
@http-too-old {
    not protocol http/2+
    not path /rss.xml /atom.xml # allow feeds
}

respond @http-too-old 400 {
    body "Due to stupid bots I have disabled http1.1. Use more modern software to access this site"
    close
}

This is quick, dirty, and will certainly need tweaking but I think it is a good enough start to see what effects it will have on my traffic.

Future Work

• Allow some bots through based on user agent. I'll probably need to do this to support things like social media previews, etc.
• Provide a better error response, redirecting to a proper page instead of a curt message.
• I should probably return 426 Upgrade Required instead of 400 but that means setting a header and the bots probably won't care.
• Redirect log lines to a different log to prevent polluting my normal access.log
• Draw a better antibot icon - the one above is poor even by my standards.