What causes a memory leak in a Java program? Shouldn't the garbage collector of the Java virtual machine (JVM) take care of unused memory? Yes, it will, but only objects that are no longer referenced can be garbage collected. A typical memory leak in Java occurs when objects that are no longer needed are still referenced somewhere in the system so that they cannot be garbage collected.

Memory leaks in Java code are a common and difficult problem to solve in large enterprise systems. These leaks are often discovered only after deployment and can be a challenge to reproduce in a testing environment. Why is that? One reason could be that the deployed system handles far larger amounts of data, and it could be that only after weeks of execution it is discovered that the Java heap is slowly increasing. Eventually this will cause the system to run out of memory.

How can we find such a leak? There are a number of tools on the market, but most of them are either based on creating a dump of the Java heap and analyzing it offline and/or based on JVMPI, which makes application execution prohibitively slow. In BEA's JVM JRockit, version 1.4.2_05, there is a technology preview of an interactive memory leak detection tool, the JRockit Memory Leak Detector, which allows usage while the system is running at almost full speed. In this article we will take a look at this experimental tool and some of its possibilities. We will also get a taste of what future versions of the tool will be capable of and go over some common types of memory leaks.

Memory Leak Detector
The Memory Leak Detector is designed for use in production environments and induces very little performance overhead on the system. A lot of the JVM-internal memory leak detection functionality is built directly into the JRockit garbage collector for higher performance. JRockit has a management console to monitor and manage one or more JVMs. The memory leak detection tool is currently built into the JRockit Management Console, but will be available as a stand-alone tool in a JRockit JDK 1.5 release later this year. The idea is that the system will help the developer understand three things: Is there a memory leak; if so, what is leaking; and where is it leaking?

Before we dig into the details, I would like to underscore that this is a low-level tool. For example, it will not tell you in what EJB your leak is. The data is passed directly from the JVM to the Memory Leak Detector and the JVM has no notion of containers, EJBs, or any other J2EE stuff. It will only deal with classes, arrays, and instances. As long as you have access to the source code, this should not pose any problems.

Approach
Let's get started. The first step is to determine if there really is a memory leak in the system. To do this we first create a connection in the management console to the JVM we are interested in, connect, and make sure the system is running and under load. Then we go to the "MemLeak Detector" tab in the management console. Here we need to enable the memory leak detection system in JRockit, which will start a trend analysis of memory growth. Each time there is a garbage collection in the JVM, JRockit sends trend data to the Memory Leak Detector that is presented in a trend analysis table (see Figure 1).

A Case Study
The trend analysis shows the most common object types on the heap (or classes if you prefer) and the rate at which memory for these types is growing. It also gives the current memory usage of the type and the number of instances. The longer the trend analysis runs, the more reliable the trend is. The types are sorted in order of growth rate and it is the types with a high growth rate that we are interested in - these are the leaking types. In the example in Figure 1 it is clear that we have three types guilty of leaking memory: DemoLeak$DemoObject, Hashtable$Entry, and Hashtable$Entry[]. I will use this example as a case study in order to explain the functionality of the JRockit Memory Leak Detector.

Where Is It Leaking?
So, now we have detected a memory leak and we also know what types of objects are leaking. In this example, it is not very far-fetched to guess that the hash table entries are holding DemoObjects, since the number of instances of these types is approximately equal. Also, it is logical that the arrays of hash table entries are also growing, as there are more and more hash table entries.

To verify that the hash table entries are holding on to the DemoObjects we can ask to see what types are pointing to DemoObject. To do this we need to freeze the updating of the trend analysis. Then we can select the DemoLeak$DemoObject row in the table and right-click to get a popup menu with an option to "Show types pointing to this type." This will display a table of all types pointing to DemoObject. In this case the table only contains the Hashtable$Entry type. (Aha, our guess was correct!) From that table it is possible to drill down and check what types are pointing to Hashtable$Entry and so on.

Traversing this "points-to chain" backwards (see Figure 2) will reveal that Hashtable$Entry types are pointed to by Hashtable$Entry[] (arrays) and also Hashtable$Entry. If you think the latter is weird, remember that in the case of a hash collision in a hash table, a single bucket stores multiple entries, which are set up as a linked list of hash table entries that is searched sequentially.

Now that we know that one or more hash tables are leaking, we want to get down to the instance level. One way is to find the Hashtable$Entry[] instance(s) that grows, which is probably the largest one. To do this we can right-click on the Hashtable$Entry[] type and ask to "Show largest arrays of this type." This action will display a table of the largest array instances and the memory size in bytes of these arrays. In our example it looks something like Listing 1.

The number in brackets <n> is an object ID that uniquely identifies the object and is used by JRockit's memory leak detection system. What this list tells us is that the array with object ID <10> is clearly a suspect as it consumes way more memory than the others. To be absolutely sure, we can wait for a while and then ask the system once again for the table of largest Hashtable$Entry arrays. We can then verify that the array with ID <10> still consumes the most memory and that it has grown a few hundred bytes.

Let's recap what we've found out so far. The system is leaking memory and there are three types that seem to be leaking - DemoObject, Hashtable$Entry, and Hashtable$Entry[]. There is one single array instance of hash table entries that is growing, so there is obviously only one hash table in the system that is leaking. Which one?

We can find out by asking for points-to information on the largest array instance by right-clicking on it and selecting "Show instances pointing to this array." This will open the window shown in Figure 3. It shows instances or static fields (if any) referring to the Hashtable$Entry[]<10> instance. As we can see, there is a single instance pointing to the array: a hash table with object ID <20>. We have now found the leaking hash table and we can continue our search by asking for points-to information for this hash table. The results are represented visually in the instance graph in Figure 4. The leaking hash table is referred by two DemoThread objects and a number of ObjectMonitors. We can disregard the ObjectMonitors because they are only a result of synchronization and have nothing to do with the application code. DemoThread on the other hand is very interesting, as it is part of the user code.

To help us quickly find the correct instance field in the DemoThread that is referring the hash table, we can open an Inspector on one of the DemoThread objects. The Inspector (see figure 5) shows all the fields of the instance and their values. The first field in the list happens to be "table," which points out a Hashtable with ID <20> that was the leaking hash table. This is how far the Memory Leak Detector gets us. Now it is simply a matter of reading the source code of DemoThread (which is an inner class in DemoLeak.java) and seeing what we are actually doing with this "table" field.

Limitations
The program we have been investigating is a simple example chosen for clarity. A real production system is a lot more complex and may need more thorough research.

You should also be aware that there are some limitations with this technology preview of the Memory Leak Detector tool. One issue is that it is now using the Java-based communication protocol of the JRockit Management Console. This means that JRockit needs to create new Java objects to send information over to the management console. This is not an ideal situation, as the system is probably already low on memory in the case of a memory leak.

Another limitation is that when there are large amounts of information to send over, we risk losing the connection to the management console because of timeout problems. This is usually only a problem when requesting a list of all instances of one kind of type pointing to a particular type (which in theory could be hundreds of thousands). The navigation in the user interface also leaves a lot to be desired.

We view this tool as a proof-of-concept and we are eager to get feedback so we can improve future versions of it. Even if it is not perfect, we believe it can still be very useful to people seeking out a memory leak.

Future Work
So, what is planned for the next version of the Memory Leak Detector? First, it will be a stand-alone tool that can be run independently from the management console. It will use a native communication protocol, which should eliminate the out-of-memory risks in the present tool. The major thing, however, will be improvements to the user interface. There will be object and type reference graphs, similar to those in Figures 2 and 4, to visualize the points-to chains and where the user can navigate in a more intuitive way than that of today. It will also be possible to get allocation traces for where a particular type of object is allocated. This can come in very handy when pinpointing exact locations in the code. The memory leak detection system in JRockit will still be operating in real time with minimal overhead and can be enabled or disabled at any time.

Common Leaks
Memory leaks are difficult to detect and fix, but the good news is that many programmers tend to make the same mistakes. That might not sound good at all, but it means that most memory leaks are quite similar. The set of typical memory leaks is therefore not that large and this will, with experience and the right tools, make them easier to find. Below, I will describe a few of these often seen on the server side.

To put objects in hash tables or hash maps but then forget to remove them when they are no longer needed is a very common error. Our case study is an example of an off-by-one error when removing items from a hash table. The faulty code in DemoThread is shown in Listing 2.

The approach here is to find out which hash table instance is leaking and who is holding onto that instance, as we did in our case study. Hopefully, this will give you enough information to pinpoint the correct hash table in your code.

A similar situation is when there is a complicated data-structure with hash tables inside other hash tables and where one or more of the "inner" tables are leaking. The approach is the same as above, but the reference information will appear more confusing, as a hash table entry can also be pointing to a hash table.

Another common leak occurs when objects are inserted in some other kind of collection, but not all of them are removed after use. Take the example of a java.util.LinkedList. In the trend analysis we will see that the LinkedList$Entry type is growing. Getting the referrers of that type will not get us very far as the type is pointing to itself - it is a linked list after all. This is a good time to use the option "Show instances of this type pointing to type " in the Memory Leak Detector, where T in this case is LinkedList$Entry. This will display a table of entry instances pointing to other entries, together with information on how much data is kept alive by each instance. This data-kept-alive number indicates how much memory the instance is keeping alive (assuming only that object was alive in the system). The entry keeping the most data alive is normally the first entry in the leaking list. Once we have found this entry we can get the instances referring to it to see where this linked list is referenced in the system. This should lead us to the right place in the code.

Summary
The JRockit JVM has some unique capabilities for real-time memory leak detection. Memory leaks in Java are often hard to find but with the right tools it is not an impossible task. I have shown how you can use the JRockit Memory Leak Detector to detect a memory leak, find out what is leaking, and then drill down to what is causing the leak in the code.

Try It Yourself
BEA WebLogic JRockit 1.4.2_05 can be downloaded free at http://commerce.bea.com/products/weblogicjrockit/1.4.2/142_05.jsp. The installation includes the management console with the built-in Memory Leak Detector. Also see http://e-docs.bea.com/wljrockit/docs142/userguide/memleak.html.