Using libcurl effectively and safely in C++

All the code here is available in my github repository along with some example code and better documentationlibcurl is an amazing project - one of the best long-running open source products around. It is so fully-featured and easy to integrate that it is easy to forget you should be very careful when including any kind of web access.

The example code floating around on the web that purports to show how to integrate libcurl with your C++ program leaves much to be desired. There is a lot that can go wrong when making HTTP requests and you are only in control of one side of the conversation. Error handling is very important.

This code is aimed only at the simple use case of downloading a simple file from a server but does it in the safest and most robust way possible. It would be easy to extend in the same style to perform POSTs or PUTs.

I think the code is pretty well commented. The one tricky feature is the way it handles exceptions in the curl callbacks:

size_t HTTPDownloader::writeFunction(char *ptr, size_t size, size_t nmemb, void *userdata)
{
    HTTPDownloader::TransferData* transfer = reinterpret_cast<HTTPDownloader::TransferData*>(userdata);
    try {
        size_t bytes = size * nmemb;

        // process the data, which may throw
        .
        .
        .

        return bytes;
    } catch (...) {
        // catch all exceptions (include those we haven't thrown ourselves - who knows what outputStream.write does?)
        // Store the exception and return an error. This is to prevent exceptions from being thrown through curl
        transfer->exception = std::current_exception();
        return 0;
    }
}

HTTPDownloader::writeFunction() is called from the internals of libcurl itself and libcurl is decidedly not exception safe. This code uses the slightly unusual technique of catching all exceptions and storing them in the TransferData structure. It then returns a value the ensures libcurl will immediately return an error to our code.

Once there we can do this, having successfully smuggled the exception out safely.

if (transferData.exception) {
  std::rethrow_exception(transferData.exception);
}

If you do use HttpDownloader, or even if you don't, I strongly advise you to do at least the following:

• Check the HTTP status returned by the server. You never know when the request will fail.
• Check the mimetype is something you want to handle. You never know when the server will decide to return something else or a middlebox will interfere.
• Limit the maximum size you are prepared to accept. You don't want to accidentally download 2Gbs worth of junk due to a bad request.
• Provide a timeout so that your program doesn't pause for minutes if a request is unexpectedly slow.

These recommendations come from having things go wrong in the field. If you ignore them then nothing bad will happen ... initially. But then it will.

Ask me how I know.

All the code is available in my github repository along with some example code and expanded documentation.