Archive for ‘Calling For Backup’

October 22nd, 2009

Source Control – Calling for Backup

by Tim Cull

This second in my Calling For Backup series is a quick argument about why source control is not optional, period.

The Scene

You’re working for a start-up in your parent’s bedroom. Development is cruising at a million miles an hour fueled by Dr. Pepper and swagger. Or you’re on a small team inside a large corporation–maybe doing skunkworks development, or maybe working on a thankless legacy application that only you (at best) know how to understand. Maybe there are one or two other developers, but largely everyone knows everyone else and the state of the code can be held in one person’s head.

Nobody on your (small) team wants to take the time to implement source control because they think it’s “too complicated” or “a hassle” or “will take too much time”. You missed the easiest chance to do it at the very beginning of your project when there was no code, so trying to do it now involves reconciling the local copies everyone has on their PCs or shared drives with everyone else’s copies.

Or maybe you do have source control, but it’s anyone’s guess is what’s deployed in production is actually the same as what’s in source control; so for all practical purposes you really don’t have source control. Again, the only way to fix the problem is to painstakingly reconcile production with source control.

“Isn’t it easier to just deploy what we know has changed? It’s worked fine so far and we have so much other ‘real work’ to do that will pay the bills,” says your boss/investor/co-hacker.

The Real Problem

Your team is suffering from a bad case of short-term-ism and poor risk estimation.

Allow yourself to imagine for a second if your primary production machine and/or your primary development machine went up in smoke. What would you have lost? How long would it take you to recover from the loss? Could you even ever be sure you did recover everything, or might you just have to cross your fingers and hope you remembered it all? All of these questions are measures of risk–and its consequences.

But the obscene risk you’re running isn’t the only problem. Try honestly reflecting back on how many decisions you’ve deferred because you can’t be sure you can undo them. Is there a big refactoring you’ve been wanting to do, but are afraid to because it’s too much change to handle all at once (what if you change your mind?). How about a new technology or library you’d like to try out, but haven’t because you can’t be sure it will work out and don’t want to manually roll it back if it sucks? These are hidden costs to not having source control.

Lastly, you are losing out on productivity gains. Imagine how much cheaper it would be to outsource your commodity development to Brazil or Ukraine, if only you wouldn’t have to give those outsourced developers access to your production hardware just to change a stylesheet or two.

Why It’s Hard to Explain

There are three reasons why source control can be a hard sell:
1) it’s surprisingly difficult for people who haven’t used it before to understand how it works
2) many of its benefits revolve around “what ifs” and “if onlys”. Your detractors will demand specific examples of what might go wrong, but by definition you can’t predict the unpredictable.
3) all of its costs are concrete and immediate, but its benefits are intangible and/or long-term.

The Rationale

Not using source control borders on professional negligence. It’s item #1 on the “Joel Test”. I doubt you’d even get beyond level 0 (hint: that level doesn’t really exist, it’s so low) in the CMMI maturity model without source control. Steve McConnell Himself would slap your grandmother if he found you without source control. But that’s all just assertion by authority. How about some concrete reasons…

If you are a software company, then the only real asset you have is your source code. Not having your source code in source control is equivalent to Walmart not locking the doors on its stores when they close: it’s just inviting a catastrophic loss. Even if you aren’t a software company per se, it’s likely software is a significant part of either your product or of the way you make your product. Without it, you would be as hosed as a retailer if the mob had robbed all its warehouses.

In addition to protecting you from loss, source control allows you to roll back. The benefits of rolling back aren’t as obvious at first, but they include:
1) Allowing you to do “spikes” to test out new ideas, without having to worry about backing them out if you change your mind
2) Helping you roll back individual features just before a release if testing shows them to be a problem
3) Allowing you to upgrade libraries without worry that you can’t go back if there are problems
4) Undoing crappy work the a new/untalented/sloppy developer has done

Lastly, source control offers you a convenient way to share code. Instead of zipping up your code and emailing it to someone, you can just point them at a URL. This is especially useful when you want to outsource development, or when you have a distributed team.

If you are having a hard time getting your team to understand the concept of source control, then Eric Sink wrote a good guide to understanding source control. Disclaimer: Eric Sink’s company sells a source control product. The nice people at Subversion also have a good primer on source control. There’s also a good MSDN article on branching and merging

Some Real Stories

My best “catastrophic loss” story is from a web development client. When I got to them, the client wasn’t using source control. In fact, they were doing development either directly on their production hardware or in Dreamweaver on their own Mac and then pushing the entire local image out to production using Dreamweaver’s FTP function.

When I came on I basically refused to work this way. I created my own Subversion repository and put all my work in there. When I needed to release, I copied the files I’d changed (which Subversion helpfully highlighed for me) to their production server by hand. This all worked reasonably well, until the client hired a second consultant in addition to me. That second consultant refused to use source control, insisting that it was “too complicated.” So from time to time we would step on each other’s changes and basically cause problems for each other. During these times, entire parts of the client’s site were broken. No amount of begging could persuade him to change.

Finally, about 6 months after I left the project, I got an email from the client. She had accidentally pushed out a bad version of the site from Dreamweaver and had basically hosed the entire site. Luckily, I was able to pull my latest version out of source control and have it restored for her in 20 minutes. But any changes that other consultant had made in the last 6 months?…..gone.

More recently, I was working on a team that had to push out a big release just after one of their senior developers had quit. When they entered user acceptance testing, they discovered that the very last thing that developer had checked in before quitting was causing lots of problems and needed to be taken out. Unfortunately, the changes were shotgunned all over the code base in tiny chunks. Luckily, they had source control to understand what he had changed and to sort out the mess and were still able to release on time. If they hadn’t had source control the only choice they would have had was simply barreling forward into the unknown.

Guerrilla Work Arounds

If you simply can’t convince your team to use source control, then at the very least do what I did and create a repository for yourself. It will only take one time when that repository saves the day (and it will, believe me) to convince the rest of the team to switch over. Both Subversion and CVS are free and will install on your local machine if necessary. But that isn’t even necessary because there are plenty of free or low cost hosting services, too.

Posted in Calling For Backup, Technology | 1 Comment »

September 8th, 2009

Using a Profiler — Calling For Backup

by Tim Cull

This first in my Calling For Backup series is a quick argument about why the very first thing anyone should do when tackling a performance problem is attach a profiler.

The Scene

You have a slow app. The team is throwing around wild ideas how to fix it. Maybe you’ve even tried a few and in spite of being expensive to implement they still didn’t make much difference. Your user community is jaded, upper management is hounding you, and you can’t sleep. A few of the more talented engineers on your team have tried to use a profiler and either couldn’t get the money to buy one or had issues trying to get it to work.

The Real Problem

You’re in guess-and-firefight mode. You need to take a step back and invest in attaching a profiler even though your first shallow pass (if any) was unsuccessful. Really, it will be well worth the investment. Your stakeholders, though, will not give you any time or budget to spend on silly things like profilers.

Why It’s Hard to Explain

By definition you can’t predict to your stakeholders ahead of time what a profiler will find or how big its impact will be. There’s a small chance it won’t find anything and you’ll look like an idiot.

The Rationale

It only takes one line of bad code to sink an app’s performance. In a 30,000 line application, there’s an excellent chance something is lurking in there.

There are a few free profilers, including the one that ships with JDK 1.6. Even paid profilers cost less than a thousand dollars for one license. A thousand dollars is equivalent to a day and a half of a single developer’s time–something you would easily eat with a single hour-long meeting with a team of ten brainstorming ideas out of thin air (the only other alternative to using a profiler).

Some Real Stories

I’ve personally profiled three separate applications that had never been profiled before and yielded big problems that had easy fixes. In one example, a bad implementation of hashCode was bringing a GUI to its knees. In a second example, the constructor of SimpleDateFormat was eating up literally half the processing time in an application’s workflow.

In a third example I found that up to 3GB of an application’s 5GB heap was being taken up with an irritating behavior in java.io.ObjectOutputStream. 5GB heap you say? Yes, for years instead of attaching a profiler (which makes this problem stick out like a sore thumb) the team had just been increasing the heap size until they didn’t run out of memory any more. But the problem is that when even the best JVM does a full garbage collection on a 5GB heap it stops the world for 90 seconds at a time.

I’ve also been lucky once where a profiler would have caught the problem sooner. The lucky time, I was working on a trading system that had a nasty memory problem. After just a short while, especially under load, the system would max out its heap and fall over dead. My boss placed a bounty on the problem. When that didn’t work, we had consultants in from the outside and locked them and our two best engineers in a room for two weeks. Even that didn’t work and finally I happened to get lucky one day and follow a strange bit of code to find the culprit, deep inside a hand-rolled logging class: the code was happily putting every single log message into a Vector that was supposed to be drained off on a socket connection. The problem was that we had long since stopped hooking up any socket listeners, so instead all the logging messages were stacking up in the Vector, living forever. Under load, of course, the log messages stacked up faster.

This last example was from the days before I’d discovered profilers. Don’t ask me why the (very expensive by the way) consultant hadn’t hooked up a profiler, but if any of us had just done it in the first place then that Vector would have stuck out like a beacon in the night–a huge, glowing sign saying “there’s your problem right there.”

Guerrilla Work Arounds

It’s possible that you’ve tried using a profiler before and just couldn’t do it. Or maybe your company won’t pay for one. It’s true that sometimes getting these guys to work is a challenge; in the 5GB heap example, the only way I could get the profiler to not choke trying to parse a heap file that big was to run the profiler in the Amazon cloud on a 64-bit Windows server.

So if you can’t get dedicated developer time from your stakeholders willingly, then this is one of the few times I’d suggest just taking it. Either add a little profiler setup to each story card or feature development, or ferret away a developer and say he’s working on “audit issues”. I promise, it will be worth it. Once you’ve improved the performance of your application by one or two orders of magnitude, nobody will mind you having taken the time.

Posted in Calling For Backup, Technology | 1 Comment »

Tags:
September 8th, 2009

Calling For Backup: A New Series

by Tim Cull

I’m starting a new series on this blog today called “Calling for Backup.”

We’ve all had those conversations, where we’re arguing something that really should be a no-brainer and, yet, the person we’re arguing with is…well…arguing. I find these situations difficult because trying to explain my reasoning is kind of like explaining why I breathe.

ThickHead: “Why do you keep breathing, it looks like so much work!”

Me: “Because if I stop I’ll die.”

ThickHead: “Nahh, I’ve never heard of anyone dying because they decided to stop breathing. Prove it!”

I’ve finally learned that a very effective tool in these situations is a story. The more real, the better. But my biggest problem is that I don’t walk around cataloging stories so I can just whip them out on demand. I doubt you do, either.

So, my series, “Calling for Backup” will be an attempt to catalog stories in a format that makes them easy to reference. So next time, I hope your conversation will be more like this:

ThickHead: “Why do you want me to attach a profiler to our system to find its performance bottlenecks? We haven’t used one in ten years and we’re fine. Besides, I’m absolutely convinced if we move from Weblogic to JBoss everything will be ten times faster.”

You: “If you haven’t attached a profiler in ten years I will bet you money you’ll find juicy, easy to fix problems.”

ThickHead: “Nah, it’s too much work. Jonny tried it once in 2003 and our app server fell over because it was too slow. Plus it will take at least, what, a full day of a developer’s time? Thanks anyway.”

You: “Here’s the thing. Tim Cull found two different problems on two different real systems that manage billions of dollars using a profiler. And they were easy to fix and there’s no way anyone would have found them without a profiler.”

ThickHead: “Oh, I see what you’re saying. Well, ok, you can have a day or so.”

Stay tuned. The first Call for Backup will be about exactly that: why you should use a profiler.

Posted in Calling For Backup, Technology | 1 Comment »

Tags: