A brief post on event driven management of Varnish Cache content. This is something that Richard Guest and I have been experimenting with off and on for a while. The concepts are similar to those covered by David Harrigan in a couple of recent blog posts (part1, part2). David's posts explain the concepts well and are worth a read. There are a few differences with our implementation that deserve a mention.
Banning Versus Purging
We want to let Varnish serve stale content when it can't reach the origin servers. We also need to be able to use wild cards in the ban expressions. This rules out HTTP purging for us - with an HTTP purge there is no option to use wild cards. Also, with a successful purge the content is gone from the cache whether the origin server is reachable or not - there is no option for Varnish to serve stale content.
We use MULE for our ESB and ActiveMQ for the messaging provider. Not an important difference, but useful to know that you can manage content with a variety of means.
I like Java most of the time, but it can be frustratingly verbose to use with HTTP. I don't want to have to write code just to add a string to the http method (e.g., BAN). The work-around is to change the Varnish config to use an HTTP method name that is available in Java.
We're going to ban content with wild cards. We have no other need for the HTTP DELETE method so we use that to implement BAN in our Varnish config. We allowed the use of an additional HTTP header (X-BanExpression) to avoid having to deal with URL encoding for regular expressions.
This means we don't need any custom implementations for MULE HTTP outbound endpoints:
The MULE project is available on Github.
We run multiple origin servers for Varnish. This means we would like redundancy in our messaging as well. The best way to achieve redundancy depends on the messaging provider. If possible, it makes sense to run a messaging provider with each origin server and use fail over in the ESB client on each Varnish server. I've covered fail over options with MULE and ActiveMQ previously.
This is a potentially powerful solution - the option for efficient caching with new content being available as soon as possible. The flexibility in implementation is also a testament to Varnish and their focus on doing one thing and doing it really well.