So we’ve been suffering with a nasty bug that caused our application server to run out of memory (1.5 GB worth, the limit you can have for a JVN running on a Windows machine) when under heavy load. The whole team had been looking for the cause for months, we hired a pair of Thoughtworks consultants to find it, and still the leak avoided detection.
Finally, by sheer accident, I found it. It turned out that deep in our logging code was a single statement that added every single log line ever written also to a Vector held in memory by each EJB. Since our application was doing a lot of logging (including logging entire FixML messages), that meant that we were holding 500MB of useless data in memory forever.
This code was apparently a hack from many years back that was necessary in order to somehow return information to our Visual Basic GUI. The functionality was no longer needed, but the dead code saving up all the logging information was still there. Talk about a prime example of dead code biting you! (and a prime example of why you shouldn’t roll your own logging code).
