A Technologist Who Speaks Business
I’ve had a good week at QConSF. I’ll try to summarize some of the themes I saw in the tracks I attended.
Many of the presenters were talking about applications at truly mind-blowing scale. Historically, that kind of scale would only apply to secret government operations and serious physics research. But today we’re talking about that scale for silly consumer sites where people post pictures about their cats. The best takeaway from that is that everyone should design with serious scale in mind and no longer assume it doesn’t appply to them. If it doesn’t apply to you today, it may very well tomorrow.
Every one of these guys that operate at scale do so on commodity or almost commodity hardware. I didn’t hear anyone mention 64-way Sun servers, except as a joke.
Applications have to be designed for asynchronous interaction. That means not only between tiers, but also with the user. To get the kinds of performance and resiliency gains many of these sessions were talking about, you can’t do it any other way. Also, asynchronicity helps take advantage of future innovations like cloud computing.
Map reduce in general and Hadoop in particular were everywhere. Even the Microsoft guy made a plug for Hadoop. If you’re a technologist and looking for a good framework to spend some time learning, you can’t go wrong with Hadoop.
I remember many years ago attending an SDWest conference where every presenter said something like “and of course here you’d run your unit tests” as if using unit tests were a foregone conclusion. That’s what measuring, monitoring, and profiling were like at this conference. If you’re not doing it routinely on your applications, then you are seriously missing the boat. (that goes double for unit testing, you slackers!)
Enterprise developers (which is most of my background) spend a lot of time and effort designing systems so that they “never” go down. If we lose a piece of hardware, or a disk drive, or a network link, then that’s a big deal and often we’ll spend many hours in meetings explaining the root cause and what steps we’re taking to make sure it never happens again. Enterprise developers could learn a lot from the huge-scale web developers who dominated this conference: just give up! Design systems to expect failure and just retry when it happens. As systems reach a certain level of complexity, it becomes literally impossible to guarantee that all their individual components are in place and healthy at the same time, all the time. Instead we have to design systems to assume failure and survive anyway. This is the true meaning of “fault tolerant” — an often abused phrase sometimes falsely interpreted to mean “never breaks.”
To listen to the presenters at this conference, you’d think ACID is dead and buried. At massive web scale, you instead have to design for ACID’s naughty younger brother: Eventual Consistency. This means that different actors using different parts of a system might get different views of the same piece of data. Eventual Consistency’s answer to that problem?: “Get over it, they’ll be the same soon enough.” As with the Expect Failure concept, I suspect this might be a hard (but necessary) pill for enterprise developers to swallow.♦ End