In-memory application caches are a fantastic way to improve application performance. In some cases, they can be done quickly and cheaply with stunning performance improvements. But in-memory caches can also kill application performance in myriad ways. For this post I’ll focus on only one: pre-loading caches on application start up.
I have worked on half a dozen different applications that pre-load caches on start up. In every one of those cases, the application server container would start up and then there would be some kind of hacky, custom-built startup class that held back client requests until some other custom code queried various data sources and filled in-memory caches with data. Only when all caches have been filled are client requests allowed to proceed. Some of these applications had huge caches–well into the gigabyte range–and took ages to warm up.
This. Is. Bad.
It’s bad for the obvious reason that it takes longer for the application server to start up and be serviceable, thereby increasing the length of development and testing cycles. Across an entire project’s lifetime, the time spent with team members waiting for caches to pre-load on startup alone can add up to hundreds of thousands of dollars.
But the obvious reason isn’t even the worst reason. The worst reason pre-loading on startup is bad is that it leads developers to assume that the cache will always be populated and that it will always be populated with the entire dataset. This assumption is a disaster for your application because:
- Then logic to detect cache misses and fetch individual items on a miss never gets written or is written badly, and as a consequence…
- Nothing can ever be evicted from the cache, which means…
- Operational teams have no options when faced with memory or data consistency issues in production, and…
- Keeping cache data synchronized with external “golden masters” is more difficult because you have to actually figure out the differences and re-query the golden source each time there’s a change instead of just invalidating the cache entry and lazily re-fetching it later, plus…
- Buggy code tends to be written that assumes literally the same object is in memory at all times and anything can be attached to it, including objects that are in other caches, which leads to…
- A tangled mess of cache inter-dependencies, which is very difficult to untangle a few years down the road when the team realizes cache pre-loading on startup is a bad idea and tries to do something about it.
“You’re just being alarmist, Tim. Those things never really happen because application teams know better than to get into that situation,” you say. Balderdash. Every, single, pre-loading application I’ve ever seen eventually falls into exactly the same scenario I just described. This is because of the Possibility Law: anything that is possible in your application’s code will eventually be done by some junior programmer somewhere. If you pre-load caches on startup, then you’ve made it possible for someone to assume caches are always full and always contain the entire data set. If you don’t pre-load, then you haven’t made that ubiquitous-data assumption possible.
But there is one draw-back to not pre-loading caches: when the application is cold and the cache is empty, warming the cache by grabbing data from the data store on a onesie-twosie basis is fantastically inefficient. I have a solution for this problem….