Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Memory Leak Detection with the JRockit JVM

Unique capabilities with the right tools

What causes a memory leak in a Java program? Shouldn't the garbage collector of the Java virtual machine (JVM) take care of unused memory? Yes, it will, but only objects that are no longer referenced can be garbage collected. A typical memory leak in Java occurs when objects that are no longer needed are still referenced somewhere in the system so that they cannot be garbage collected.

Memory leaks in Java code are a common and difficult problem to solve in large enterprise systems. These leaks are often discovered only after deployment and can be a challenge to reproduce in a testing environment. Why is that? One reason could be that the deployed system handles far larger amounts of data, and it could be that only after weeks of execution it is discovered that the Java heap is slowly increasing. Eventually this will cause the system to run out of memory.

How can we find such a leak? There are a number of tools on the market, but most of them are either based on creating a dump of the Java heap and analyzing it offline and/or based on JVMPI, which makes application execution prohibitively slow. In BEA's JVM JRockit, version 1.4.2_05, there is a technology preview of an interactive memory leak detection tool, the JRockit Memory Leak Detector, which allows usage while the system is running at almost full speed. In this article we will take a look at this experimental tool and some of its possibilities. We will also get a taste of what future versions of the tool will be capable of and go over some common types of memory leaks.

Memory Leak Detector
The Memory Leak Detector is designed for use in production environments and induces very little performance overhead on the system. A lot of the JVM-internal memory leak detection functionality is built directly into the JRockit garbage collector for higher performance. JRockit has a management console to monitor and manage one or more JVMs. The memory leak detection tool is currently built into the JRockit Management Console, but will be available as a stand-alone tool in a JRockit JDK 1.5 release later this year. The idea is that the system will help the developer understand three things: Is there a memory leak; if so, what is leaking; and where is it leaking?

Before we dig into the details, I would like to underscore that this is a low-level tool. For example, it will not tell you in what EJB your leak is. The data is passed directly from the JVM to the Memory Leak Detector and the JVM has no notion of containers, EJBs, or any other J2EE stuff. It will only deal with classes, arrays, and instances. As long as you have access to the source code, this should not pose any problems.

Approach
Let's get started. The first step is to determine if there really is a memory leak in the system. To do this we first create a connection in the management console to the JVM we are interested in, connect, and make sure the system is running and under load. Then we go to the "MemLeak Detector" tab in the management console. Here we need to enable the memory leak detection system in JRockit, which will start a trend analysis of memory growth. Each time there is a garbage collection in the JVM, JRockit sends trend data to the Memory Leak Detector that is presented in a trend analysis table (see Figure 1).

A Case Study
The trend analysis shows the most common object types on the heap (or classes if you prefer) and the rate at which memory for these types is growing. It also gives the current memory usage of the type and the number of instances. The longer the trend analysis runs, the more reliable the trend is. The types are sorted in order of growth rate and it is the types with a high growth rate that we are interested in - these are the leaking types. In the example in Figure 1 it is clear that we have three types guilty of leaking memory: DemoLeak$DemoObject, Hashtable$Entry, and Hashtable$Entry[]. I will use this example as a case study in order to explain the functionality of the JRockit Memory Leak Detector.

Where Is It Leaking?
So, now we have detected a memory leak and we also know what types of objects are leaking. In this example, it is not very far-fetched to guess that the hash table entries are holding DemoObjects, since the number of instances of these types is approximately equal. Also, it is logical that the arrays of hash table entries are also growing, as there are more and more hash table entries.

To verify that the hash table entries are holding on to the DemoObjects we can ask to see what types are pointing to DemoObject. To do this we need to freeze the updating of the trend analysis. Then we can select the DemoLeak$DemoObject row in the table and right-click to get a popup menu with an option to "Show types pointing to this type." This will display a table of all types pointing to DemoObject. In this case the table only contains the Hashtable$Entry type. (Aha, our guess was correct!) From that table it is possible to drill down and check what types are pointing to Hashtable$Entry and so on.

Traversing this "points-to chain" backwards (see Figure 2) will reveal that Hashtable$Entry types are pointed to by Hashtable$Entry[] (arrays) and also Hashtable$Entry. If you think the latter is weird, remember that in the case of a hash collision in a hash table, a single bucket stores multiple entries, which are set up as a linked list of hash table entries that is searched sequentially.

Now that we know that one or more hash tables are leaking, we want to get down to the instance level. One way is to find the Hashtable$Entry[] instance(s) that grows, which is probably the largest one. To do this we can right-click on the Hashtable$Entry[] type and ask to "Show largest arrays of this type." This action will display a table of the largest array instances and the memory size in bytes of these arrays. In our example it looks something like Listing 1.

The number in brackets <n> is an object ID that uniquely identifies the object and is used by JRockit's memory leak detection system. What this list tells us is that the array with object ID <10> is clearly a suspect as it consumes way more memory than the others. To be absolutely sure, we can wait for a while and then ask the system once again for the table of largest Hashtable$Entry arrays. We can then verify that the array with ID <10> still consumes the most memory and that it has grown a few hundred bytes.

Let's recap what we've found out so far. The system is leaking memory and there are three types that seem to be leaking - DemoObject, Hashtable$Entry, and Hashtable$Entry[]. There is one single array instance of hash table entries that is growing, so there is obviously only one hash table in the system that is leaking. Which one?

We can find out by asking for points-to information on the largest array instance by right-clicking on it and selecting "Show instances pointing to this array." This will open the window shown in Figure 3. It shows instances or static fields (if any) referring to the Hashtable$Entry[]<10> instance. As we can see, there is a single instance pointing to the array: a hash table with object ID <20>. We have now found the leaking hash table and we can continue our search by asking for points-to information for this hash table. The results are represented visually in the instance graph in Figure 4. The leaking hash table is referred by two DemoThread objects and a number of ObjectMonitors. We can disregard the ObjectMonitors because they are only a result of synchronization and have nothing to do with the application code. DemoThread on the other hand is very interesting, as it is part of the user code.

To help us quickly find the correct instance field in the DemoThread that is referring the hash table, we can open an Inspector on one of the DemoThread objects. The Inspector (see figure 5) shows all the fields of the instance and their values. The first field in the list happens to be "table," which points out a Hashtable with ID <20> that was the leaking hash table. This is how far the Memory Leak Detector gets us. Now it is simply a matter of reading the source code of DemoThread (which is an inner class in DemoLeak.java) and seeing what we are actually doing with this "table" field.

Limitations
The program we have been investigating is a simple example chosen for clarity. A real production system is a lot more complex and may need more thorough research.

You should also be aware that there are some limitations with this technology preview of the Memory Leak Detector tool. One issue is that it is now using the Java-based communication protocol of the JRockit Management Console. This means that JRockit needs to create new Java objects to send information over to the management console. This is not an ideal situation, as the system is probably already low on memory in the case of a memory leak.

Another limitation is that when there are large amounts of information to send over, we risk losing the connection to the management console because of timeout problems. This is usually only a problem when requesting a list of all instances of one kind of type pointing to a particular type (which in theory could be hundreds of thousands). The navigation in the user interface also leaves a lot to be desired.

We view this tool as a proof-of-concept and we are eager to get feedback so we can improve future versions of it. Even if it is not perfect, we believe it can still be very useful to people seeking out a memory leak.

Future Work
So, what is planned for the next version of the Memory Leak Detector? First, it will be a stand-alone tool that can be run independently from the management console. It will use a native communication protocol, which should eliminate the out-of-memory risks in the present tool. The major thing, however, will be improvements to the user interface. There will be object and type reference graphs, similar to those in Figures 2 and 4, to visualize the points-to chains and where the user can navigate in a more intuitive way than that of today. It will also be possible to get allocation traces for where a particular type of object is allocated. This can come in very handy when pinpointing exact locations in the code. The memory leak detection system in JRockit will still be operating in real time with minimal overhead and can be enabled or disabled at any time.

Common Leaks
Memory leaks are difficult to detect and fix, but the good news is that many programmers tend to make the same mistakes. That might not sound good at all, but it means that most memory leaks are quite similar. The set of typical memory leaks is therefore not that large and this will, with experience and the right tools, make them easier to find. Below, I will describe a few of these often seen on the server side.

To put objects in hash tables or hash maps but then forget to remove them when they are no longer needed is a very common error. Our case study is an example of an off-by-one error when removing items from a hash table. The faulty code in DemoThread is shown in Listing 2.

The approach here is to find out which hash table instance is leaking and who is holding onto that instance, as we did in our case study. Hopefully, this will give you enough information to pinpoint the correct hash table in your code.

A similar situation is when there is a complicated data-structure with hash tables inside other hash tables and where one or more of the "inner" tables are leaking. The approach is the same as above, but the reference information will appear more confusing, as a hash table entry can also be pointing to a hash table.

Another common leak occurs when objects are inserted in some other kind of collection, but not all of them are removed after use. Take the example of a java.util.LinkedList. In the trend analysis we will see that the LinkedList$Entry type is growing. Getting the referrers of that type will not get us very far as the type is pointing to itself - it is a linked list after all. This is a good time to use the option "Show instances of this type pointing to type " in the Memory Leak Detector, where T in this case is LinkedList$Entry. This will display a table of entry instances pointing to other entries, together with information on how much data is kept alive by each instance. This data-kept-alive number indicates how much memory the instance is keeping alive (assuming only that object was alive in the system). The entry keeping the most data alive is normally the first entry in the leaking list. Once we have found this entry we can get the instances referring to it to see where this linked list is referenced in the system. This should lead us to the right place in the code.

Summary
The JRockit JVM has some unique capabilities for real-time memory leak detection. Memory leaks in Java are often hard to find but with the right tools it is not an impossible task. I have shown how you can use the JRockit Memory Leak Detector to detect a memory leak, find out what is leaking, and then drill down to what is causing the leak in the code.

Try It Yourself
BEA WebLogic JRockit 1.4.2_05 can be downloaded free at http://commerce.bea.com/products/weblogicjrockit/1.4.2/142_05.jsp. The installation includes the management console with the built-in Memory Leak Detector. Also see http://e-docs.bea.com/wljrockit/docs142/userguide/memleak.html.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
High-velocity engineering teams are applying not only continuous delivery processes, but also lessons in experimentation from established leaders like Amazon, Netflix, and Facebook. These companies have made experimentation a foundation for their release processes, allowing them to try out major feature releases and redesigns within smaller groups before making them broadly available. In his session at 21st Cloud Expo, Brian Lucas, Senior Staff Engineer at Optimizely, will discuss how by using...
In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lead...
SYS-CON Events announced today that Daiya Industry will exhibit at the Japanese Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ruby Development Inc. builds new services in short period of time and provides a continuous support of those services based on Ruby on Rails. For more information, please visit https://github.com/RubyDevInc.
As businesses evolve, they need technology that is simple to help them succeed today and flexible enough to help them build for tomorrow. Chrome is fit for the workplace of the future — providing a secure, consistent user experience across a range of devices that can be used anywhere. In her session at 21st Cloud Expo, Vidya Nagarajan, a Senior Product Manager at Google, will take a look at various options as to how ChromeOS can be leveraged to interact with people on the devices, and formats th...
SYS-CON Events announced today that Yuasa System will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Yuasa System is introducing a multi-purpose endurance testing system for flexible displays, OLED devices, flexible substrates, flat cables, and films in smartphones, wearables, automobiles, and healthcare.
SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Taica manufacturers Alpha-GEL brand silicone components and materials, which maintain outstanding performance over a wide temperature range -40C to +200C. For more information, visit http://www.taica.co.jp/english/.
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
Organizations do not need a Big Data strategy; they need a business strategy that incorporates Big Data. Most organizations lack a road map for using Big Data to optimize key business processes, deliver a differentiated customer experience, or uncover new business opportunities. They do not understand what’s possible with respect to integrating Big Data into the business model.
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, will discuss how they b...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...