We developers and technologists are obsessed with abstractions and generalizations. Tell any developer they will be implementing a series of business rules for an insurance company, and they’ll roll their eyes. Tell the same developer they will be implementing a generic rules framework that lets business users specify any rule they want without any coding and they will start to salivate.
This obsession with abstraction and generalization is usually a good thing. It drives us to see patterns that others don’t see and invent new ways of thinking that drive the productivity improvements that keep our economy booming.
But there is a limit to it how useful generalization can be. That limit comes when it slams up against what I’ll call the Law of Preservation of Complexity. My Law of Preservation of Complexity says: “There is only so much of a problem that can be generalized away. Once the theoretical maximum generalization has been achieved, what remains is the fundamental problem that must be solved through brute force. Any attempt to further generalize will actually be adding complexity or obscuring the problem.”
Imagine the case of squeezing a semi-inflated water balloon. The water inside is the fundamental complexity of a problem and the shape of the balloon is how you’ve chosen to generalize the problem. If you squeeze one end of the balloon to make it smaller, then the other end of the balloon will get bigger. If you poke the other end with your finger, then the sides of the balloon will get bigger. No matter what you do to the balloon, it will still hold the same amount of water.
Now imagine this concept in a system. Say someone designed a system with data storage that’s nothing more than one giant table of name-value pairs. “This is fantastic!” the architect will say, “we can change the data model without having to change the database!” True and not true. The fact is there’s a fundamental complexity (relationships between entities that could have been represented with tables and foreign keys) that has simply been pushed up into the application layer. The complexity did not go away, it was simply moved somewhere else, possibly into a technology that’s less adept at handling it. In the worst case scenario, the complexity was moved to more than one place; it was shot-gunned across the system at random instead of compartmentalized in one, logical place.
Whenever I have a team proposing an abstraction or a generalization to me, I ask, “is this really removing complexity all together, or is it simply pushing it around, like a toddler would do with lima beans he doesn’t want to eat on his plate?” Only if the proposal passes that test do I recommend proceeding.
So if some complexity is inevitable, then what should we do with it? I recommend compartmentalizing it in one place. Put it behind an iron-clad facade with a well defined interface. Behind the interface, be as ugly and brute-force as necessary, but beyond the interface keep the system elegant and generalized. Then at least the complexity is obvious, discoverable, and easier to maintain.
September 14, 2010 at 8:31 am
This is a big topic and you strike to the heart of it. Your example of the column database is spot-on. Programmers are trained to generalize to the point where the highest level of abstraction solves every conceivable solution. The “framework” is the ultimate expression of this: whatever service or resource the application requires leads to it being coded in the framework complete with a messy API. We have to recognize when an algorithm or data structure is unique to one area of the application and should be located there, coded as itself, not as a do-everything abstraction.