Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Memory Leak Detection with the JRockit JVM

Unique capabilities with the right tools

What causes a memory leak in a Java program? Shouldn't the garbage collector of the Java virtual machine (JVM) take care of unused memory? Yes, it will, but only objects that are no longer referenced can be garbage collected. A typical memory leak in Java occurs when objects that are no longer needed are still referenced somewhere in the system so that they cannot be garbage collected.

Memory leaks in Java code are a common and difficult problem to solve in large enterprise systems. These leaks are often discovered only after deployment and can be a challenge to reproduce in a testing environment. Why is that? One reason could be that the deployed system handles far larger amounts of data, and it could be that only after weeks of execution it is discovered that the Java heap is slowly increasing. Eventually this will cause the system to run out of memory.

How can we find such a leak? There are a number of tools on the market, but most of them are either based on creating a dump of the Java heap and analyzing it offline and/or based on JVMPI, which makes application execution prohibitively slow. In BEA's JVM JRockit, version 1.4.2_05, there is a technology preview of an interactive memory leak detection tool, the JRockit Memory Leak Detector, which allows usage while the system is running at almost full speed. In this article we will take a look at this experimental tool and some of its possibilities. We will also get a taste of what future versions of the tool will be capable of and go over some common types of memory leaks.

Memory Leak Detector
The Memory Leak Detector is designed for use in production environments and induces very little performance overhead on the system. A lot of the JVM-internal memory leak detection functionality is built directly into the JRockit garbage collector for higher performance. JRockit has a management console to monitor and manage one or more JVMs. The memory leak detection tool is currently built into the JRockit Management Console, but will be available as a stand-alone tool in a JRockit JDK 1.5 release later this year. The idea is that the system will help the developer understand three things: Is there a memory leak; if so, what is leaking; and where is it leaking?

Before we dig into the details, I would like to underscore that this is a low-level tool. For example, it will not tell you in what EJB your leak is. The data is passed directly from the JVM to the Memory Leak Detector and the JVM has no notion of containers, EJBs, or any other J2EE stuff. It will only deal with classes, arrays, and instances. As long as you have access to the source code, this should not pose any problems.

Let's get started. The first step is to determine if there really is a memory leak in the system. To do this we first create a connection in the management console to the JVM we are interested in, connect, and make sure the system is running and under load. Then we go to the "MemLeak Detector" tab in the management console. Here we need to enable the memory leak detection system in JRockit, which will start a trend analysis of memory growth. Each time there is a garbage collection in the JVM, JRockit sends trend data to the Memory Leak Detector that is presented in a trend analysis table (see Figure 1).

A Case Study
The trend analysis shows the most common object types on the heap (or classes if you prefer) and the rate at which memory for these types is growing. It also gives the current memory usage of the type and the number of instances. The longer the trend analysis runs, the more reliable the trend is. The types are sorted in order of growth rate and it is the types with a high growth rate that we are interested in - these are the leaking types. In the example in Figure 1 it is clear that we have three types guilty of leaking memory: DemoLeak$DemoObject, Hashtable$Entry, and Hashtable$Entry[]. I will use this example as a case study in order to explain the functionality of the JRockit Memory Leak Detector.

Where Is It Leaking?
So, now we have detected a memory leak and we also know what types of objects are leaking. In this example, it is not very far-fetched to guess that the hash table entries are holding DemoObjects, since the number of instances of these types is approximately equal. Also, it is logical that the arrays of hash table entries are also growing, as there are more and more hash table entries.

To verify that the hash table entries are holding on to the DemoObjects we can ask to see what types are pointing to DemoObject. To do this we need to freeze the updating of the trend analysis. Then we can select the DemoLeak$DemoObject row in the table and right-click to get a popup menu with an option to "Show types pointing to this type." This will display a table of all types pointing to DemoObject. In this case the table only contains the Hashtable$Entry type. (Aha, our guess was correct!) From that table it is possible to drill down and check what types are pointing to Hashtable$Entry and so on.

Traversing this "points-to chain" backwards (see Figure 2) will reveal that Hashtable$Entry types are pointed to by Hashtable$Entry[] (arrays) and also Hashtable$Entry. If you think the latter is weird, remember that in the case of a hash collision in a hash table, a single bucket stores multiple entries, which are set up as a linked list of hash table entries that is searched sequentially.

Now that we know that one or more hash tables are leaking, we want to get down to the instance level. One way is to find the Hashtable$Entry[] instance(s) that grows, which is probably the largest one. To do this we can right-click on the Hashtable$Entry[] type and ask to "Show largest arrays of this type." This action will display a table of the largest array instances and the memory size in bytes of these arrays. In our example it looks something like Listing 1.

The number in brackets <n> is an object ID that uniquely identifies the object and is used by JRockit's memory leak detection system. What this list tells us is that the array with object ID <10> is clearly a suspect as it consumes way more memory than the others. To be absolutely sure, we can wait for a while and then ask the system once again for the table of largest Hashtable$Entry arrays. We can then verify that the array with ID <10> still consumes the most memory and that it has grown a few hundred bytes.

Let's recap what we've found out so far. The system is leaking memory and there are three types that seem to be leaking - DemoObject, Hashtable$Entry, and Hashtable$Entry[]. There is one single array instance of hash table entries that is growing, so there is obviously only one hash table in the system that is leaking. Which one?

We can find out by asking for points-to information on the largest array instance by right-clicking on it and selecting "Show instances pointing to this array." This will open the window shown in Figure 3. It shows instances or static fields (if any) referring to the Hashtable$Entry[]<10> instance. As we can see, there is a single instance pointing to the array: a hash table with object ID <20>. We have now found the leaking hash table and we can continue our search by asking for points-to information for this hash table. The results are represented visually in the instance graph in Figure 4. The leaking hash table is referred by two DemoThread objects and a number of ObjectMonitors. We can disregard the ObjectMonitors because they are only a result of synchronization and have nothing to do with the application code. DemoThread on the other hand is very interesting, as it is part of the user code.

To help us quickly find the correct instance field in the DemoThread that is referring the hash table, we can open an Inspector on one of the DemoThread objects. The Inspector (see figure 5) shows all the fields of the instance and their values. The first field in the list happens to be "table," which points out a Hashtable with ID <20> that was the leaking hash table. This is how far the Memory Leak Detector gets us. Now it is simply a matter of reading the source code of DemoThread (which is an inner class in DemoLeak.java) and seeing what we are actually doing with this "table" field.

The program we have been investigating is a simple example chosen for clarity. A real production system is a lot more complex and may need more thorough research.

You should also be aware that there are some limitations with this technology preview of the Memory Leak Detector tool. One issue is that it is now using the Java-based communication protocol of the JRockit Management Console. This means that JRockit needs to create new Java objects to send information over to the management console. This is not an ideal situation, as the system is probably already low on memory in the case of a memory leak.

Another limitation is that when there are large amounts of information to send over, we risk losing the connection to the management console because of timeout problems. This is usually only a problem when requesting a list of all instances of one kind of type pointing to a particular type (which in theory could be hundreds of thousands). The navigation in the user interface also leaves a lot to be desired.

We view this tool as a proof-of-concept and we are eager to get feedback so we can improve future versions of it. Even if it is not perfect, we believe it can still be very useful to people seeking out a memory leak.

Future Work
So, what is planned for the next version of the Memory Leak Detector? First, it will be a stand-alone tool that can be run independently from the management console. It will use a native communication protocol, which should eliminate the out-of-memory risks in the present tool. The major thing, however, will be improvements to the user interface. There will be object and type reference graphs, similar to those in Figures 2 and 4, to visualize the points-to chains and where the user can navigate in a more intuitive way than that of today. It will also be possible to get allocation traces for where a particular type of object is allocated. This can come in very handy when pinpointing exact locations in the code. The memory leak detection system in JRockit will still be operating in real time with minimal overhead and can be enabled or disabled at any time.

Common Leaks
Memory leaks are difficult to detect and fix, but the good news is that many programmers tend to make the same mistakes. That might not sound good at all, but it means that most memory leaks are quite similar. The set of typical memory leaks is therefore not that large and this will, with experience and the right tools, make them easier to find. Below, I will describe a few of these often seen on the server side.

To put objects in hash tables or hash maps but then forget to remove them when they are no longer needed is a very common error. Our case study is an example of an off-by-one error when removing items from a hash table. The faulty code in DemoThread is shown in Listing 2.

The approach here is to find out which hash table instance is leaking and who is holding onto that instance, as we did in our case study. Hopefully, this will give you enough information to pinpoint the correct hash table in your code.

A similar situation is when there is a complicated data-structure with hash tables inside other hash tables and where one or more of the "inner" tables are leaking. The approach is the same as above, but the reference information will appear more confusing, as a hash table entry can also be pointing to a hash table.

Another common leak occurs when objects are inserted in some other kind of collection, but not all of them are removed after use. Take the example of a java.util.LinkedList. In the trend analysis we will see that the LinkedList$Entry type is growing. Getting the referrers of that type will not get us very far as the type is pointing to itself - it is a linked list after all. This is a good time to use the option "Show instances of this type pointing to type " in the Memory Leak Detector, where T in this case is LinkedList$Entry. This will display a table of entry instances pointing to other entries, together with information on how much data is kept alive by each instance. This data-kept-alive number indicates how much memory the instance is keeping alive (assuming only that object was alive in the system). The entry keeping the most data alive is normally the first entry in the leaking list. Once we have found this entry we can get the instances referring to it to see where this linked list is referenced in the system. This should lead us to the right place in the code.

The JRockit JVM has some unique capabilities for real-time memory leak detection. Memory leaks in Java are often hard to find but with the right tools it is not an impossible task. I have shown how you can use the JRockit Memory Leak Detector to detect a memory leak, find out what is leaking, and then drill down to what is causing the leak in the code.

Try It Yourself
BEA WebLogic JRockit 1.4.2_05 can be downloaded free at http://commerce.bea.com/products/weblogicjrockit/1.4.2/142_05.jsp. The installation includes the management console with the built-in Memory Leak Detector. Also see http://e-docs.bea.com/wljrockit/docs142/userguide/memleak.html.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

@ThingsExpo Stories
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
DXWorldEXPO LLC announced today that ICOHOLDER named "Media Sponsor" of Miami Blockchain Event by FinTechEXPO. ICOHOLDER give you detailed information and help the community to invest in the trusty projects. Miami Blockchain Event by FinTechEXPO has opened its Call for Papers. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Miami Blockchain Event by FinTechEXPO also offers s...
Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.
Predicting the future has never been more challenging - not because of the lack of data but because of the flood of ungoverned and risk laden information. Microsoft states that 2.5 exabytes of data are created every day. Expectations and reliance on data are being pushed to the limits, as demands around hybrid options continue to grow.
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
As IoT continues to increase momentum, so does the associated risk. Secure Device Lifecycle Management (DLM) is ranked as one of the most important technology areas of IoT. Driving this trend is the realization that secure support for IoT devices provides companies the ability to deliver high-quality, reliable, secure offerings faster, create new revenue streams, and reduce support costs, all while building a competitive advantage in their markets. In this session, we will use customer use cases...
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
DXWorldEXPO | CloudEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
The IoT Will Grow: In what might be the most obvious prediction of the decade, the IoT will continue to expand next year, with more and more devices coming online every single day. What isn’t so obvious about this prediction: where that growth will occur. The retail, healthcare, and industrial/supply chain industries will likely see the greatest growth. Forrester Research has predicted the IoT will become “the backbone” of customer value as it continues to grow. It is no surprise that retail is ...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...