One of the formative books of my youth was The Cuckoo's Egg by Clifford Stoll in which Stoll famously tells the tale of how tracking down a 75 cent accounting error on a Unix server he was administrating led to an international spy ring. It is a cracking read that I recommend to anyone.
I mention it because yesterday I found myself in a similar situation. This website was running very slowly so I decided to check the logs. I discovered that a misbehaving bot was very aggressively spidering the site but in the process also noticed a persistent 10Kbps level of external outgoing network traffic even when the webserver was not running.
This is not a lot in the great scheme of things but sheep.horse is a very simple site. What could it be?
Unlike Stoll, I am not much of a sysadmin but I recognize fuckary when I see it.
My server was consorting with all sorts of mysterious boxes, mostly located in Brazil and Russia. I suspect was probably conscripted into a botnet or maybe being used as a VPN tunnel or something. It really doesn't matter why.
I shut down the whole droplet as soon as I figured out what was happening. As far as I can tell, sheep.horse wan't serving malicious files to legitimate visitors but why take the risk?
How did this happen?
Well, the droplet that sheep.horse was running on was 9 years old running an older kernel. I did keep it updated and follow basic best practices but obviously not well enough. I can't even say how long ago I first got hacked but the traffic appears on graphs 14 days old so at least two weeks.
It was time to nuke the old sheep.horse from orbit and start again with a more modern kernel. After a few hours of futzing around I got most things working again on a new droplet - you are using it now. The one thing that isn't working is the visitor statistics but that can wait.
The moral is that I need to be more vigilant in updating my server. But also that there are a bunch of people out there that ruin for everyone what should be a fun, open network.
Update: I have reinstated the visitor statistics, something that only I care about.