Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Analyzing Java Application Problems

Using Java thread dumps to assess the problem

This article discusses troubleshooting techniques for Java applications by analyzing Java thread dumps. We can use thread dumps to analyze situations like application hang, poor application response times, and application crash. Before getting into the details of analyzing the thread dumps, let's look briefly at the thread dump itself.

A Java thread dump is a snapshot of all the Java threads in a running Java application. It shows information like the current stack trace, state, and name of the threads. The list of threads includes the threads created by the JVM itself (to do housekeeping work like garbage collection, signal handling, etc.) and the threads created by the application.

You can get a thread dump by sending a SIGQUIT signal to the JVM. In case of Unix operating systems (Solaris/Linux/HP-Unix etc), this can be done by the "kill -3 <pid>" command. In the case of Windows, you can do a "ctrl-break" in the command window to get the thread dump. The thread dump is printed to the stdout or stderr of the JVM. The application will continue running normally after printing the dump. When you send the SIGQUIT signal to the JVM, the signal handler of the JVM responds to this signal by printing the thread dump. You can take thread dumps at any point while the application is running.

A Sample Thread Dump
Listing 1 is a sample thread dump from a single-threaded application that is using the Sun JVM 1.4.1. The thread named "main" is the main application thread. All other threads are created by the JVM to do housekeeping work. While analyzing application-level problems, we generally focus only on the application threads. Let's analyze the stack trace of the "main" thread from Listing 1

"main" prio=5 tid=0x002358B8 nid=0x7f8 runnable [6f000..6fc40]
at test.method1(test.java:10)
at test.main(test.java:5)

From this code snippet you can see that a thread stack trace has a name, thread priority (prio=5), state (runnable), source code line numbers, and the method calls. You can conclude from this stack trace that the thread "main" is executing some code in the method "method1" of class "test". The call to this method came from the method "main" of the same class. You can also see the exact source code line numbers in those methods.

Before moving on to analyzing thread dumps from more complex scenarios, let's discuss the different state of threads that we normally see in the thread dumps and what they mean.

  • Runnable: Either running or ready to run when it gets its CPU turn. JRockit thread dumps refer to this state as ACTIVE.
  • Waiting on monitor: Either sleeping or waiting on an object for said period of time or waiting to be notified by another thread. This will happen when any of the sleep() methods on Thread object or wait() method on an Object is called.

    For example, in BEA WebLogic Server the idle execute threads are in this condition and they wait until a socket reader thread notifies them of some new work to be done. The stack trace will look like this:

    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'"
    daemon prio=5 tid=0x1752F040 nid=0x180c in Object.wait()
    [1887f000..1887fd8c]
    at java.lang.Object.wait(Native Method)
    waiting on <04134D98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:426)

    Some other versions of the JVM call this state CW. JRockit and refers to this state as WAITING.

  • Waiting for monitor entry: Waiting to lock an object (some other thread may be holding the lock). This happens if two or more threads try to execute some synchronized code blocks/methods of an object at the same time. Please note that the lock is always for an object and not for individual methods. That is, if a thread has to execute a synchronized method of an object, it has to first lock that object.
The following is a sample stack trace of a thread in this condition:

"ExecuteThread: '24' for queue: 'DisplayExecuteQueue'" daemon prio=5
tid=0x5541b0 nid=0x3b waiting for monitor entry [49b7f000..49b7fc24]
at
weblogic.cluster.replication.ReplicationManager.createSecondary
(ReplicationManager.java:908)
- waiting to lock <6c4b9130> (a java.lang.Object)
at
weblogic.cluster.replication.ReplicationManager.updateSecondary
(ReplicationManager.java:715)

In this snippet you can see that this thread holds the lock on an object (6c408700) and is waiting to lock another object (6c4b9130).

Some other version of a JVM may not give the object IDs lock information in the stack trace. In those cases, we can guess that the thread is waiting to lock the object from the state of the thread. The same state may also be called as "MW". JRockit refers to this state as LOCKED.

Now, let's see some thread dumps from different scenarios and analyze them to find the problem in the application.

Thread Dump from a Deadlocked Application
In Listing 2 (only part of the thread dump is shown), you can see that the JVM has found the deadlock and it has given that information along with the thread dump. It clearly says that the "ExecuteThread: '47' for queue: 'default'" is waiting to lock an object that has been locked by "ExecuteThread: '57' for queue: 'default'". (This thread dump is from a WebLogic Server. These threads are WebLogic Server's worker threads, which process the client requests.) At the same time, ExecuteThread: '47' has locked an object for which ExecuteThread: '57' is waiting. This is a deadlock. Both the threads will never move from this state, waiting for the other thread to release the lock.

In Listing 2, Execute thread: 1 is waiting to lock the "ConnectionScavanger" object, but this has been locked by thread 47, which is in a deadlock with thread 57 as mentioned before. Therefore, Execute thread: 1 also won't move further. This will eventually result in a hang because the other execute threads in the server will also try to get the lock for the same objects locked by Execute thread: 47 and 57 at some later time.

In some JVMs, the JVM will not give the deadlock information. It will just give the thread dump. In that case, we can arrive at this same conclusion by looking at the state and stack trace of the threads.

In this case the problem was due to a bug in WebLogic that was later fixed.

Other Hang Scenarios
In some scenarios, you may see that most of the threads are "runnable" but the server may appear hung and won't respond to client requests. If the threads are doing IO, they may be blocked in read() - the state will be "runnable" (database or some other network response may be poor). If this is the case, you need to check the health of the database and network.

Other reasons for a "runnable" thread to be stuck at the same place could be:

  • An infinite looping in the code. This may also lead to high CPU usage.
  • JVM garbage collection may be running for a long time, taking most of the CPU.
  • The JVM process might have run out of file descriptors.
If the application code calls wait() or sleep(), the threads won't move for the specified sleep time - the state of the threads in this case will be "Waiting on Monitor". In this scenario, check the application code.

A lot of contention for a single object may also slow down the performance of the system, which may eventually lead to a hang-like situation. In this scenario, the state of most of the threads is "Waiting for monitor entry". The application code needs to be investigated to reduce the contention on that particular object. The threads may also be waiting to get a response from another server and the other server may be hung for a different reason and it may not be able to send the response to this server.

In all of the above scenarios, it is also required that you take multiple thread dumps with a few seconds interval so that you can compare the state of a particular thread in all the thread dumps and you can decide whether the thread is moving or stuck.

JVM Crash Scenarios
A JVM crash can happen due to:

  1. Bugs in the JVM library
  2. Incorrect use of JNI APIs by the application
  3. Bugs in the native modules used by the application - native database drivers, etc.
Normally, when the JVM crashes it gives the stack trace of the Java thread that caused the crash. Some JVMs give a complete thread dump (stack trace of all the threads) before exiting. Our interest in those thread dumps is to find which thread was running when the crash happened. This thread is called as a "current thread". If the JVM gives a complete thread dump, it marks the "current thread". Therefore, generally it's easy to find the thread that caused the crash.

Note that the crash dump usually goes to stdout/stderr. The JVM may also generate a core file, which will be in binary format.

Sample Crash Dump
If you look at the crash dump in Listing 3 you can see that the thread that caused the crash has the following stack trace:

Current Java thread:
at com.aaa.bbb.qqq.Direct.setString(Native Method)
at com.aaa.bbb.qqq.PreparedStatement.setString(PreparedStatement.java:51)
- locked <0x45029c60> (a com.aaa.bbb.qqq.PreparedStatement)

The last method executed (setString) was a "native method" in the class "com.aaa.bbb.qqq.Direct". Therefore, we can guess that this crash is due to the native code in the implementation of the setString method. It could also be due to the JVM library code that has the implementation for the JNI API.

You can also see that the JVM gave the following in the dump:

Unexpected Signal : 11 occurred at PC=0x403AC9D1
Function=(null)+0x403AC9D1
Library=/opt/sunjdk/1.4.1_01-b01/jre/lib/i386/client/libjvm.so
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002E6
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi

This means that the code that caused the crash (PC=0x403AC9D1) is part of the JVM library: "/opt/sunjdk/1.4.1_01-b01/jre/lib/i386/client/libjvm.so". This doesn't mean that this is necessarily a JVM bug. Even though the instruction that caused the crash is in the JVM code, the crash could be due to a JNI call made by the application code (in this case, the implementation of the native method "setString") that could have passed some incorrect arguments, which caused the crash in the JVM module.

If you don't have the "current thread" info in the dump, then we need to analyze core dumps. This is out of the scope of this document.

More Stories By Sathish Santhanam

Sathish Santhanam is a developer relations engineer in the Weblogic/JRockit backline support team. He has more than six years of experience in software development, testing, and support; is a Sun-certified Java programmer and BEA certified developer; and is an expert in debugging server hang, performance, and JVM crash problems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
Organizations do not need a Big Data strategy; they need a business strategy that incorporates Big Data. Most organizations lack a road map for using Big Data to optimize key business processes, deliver a differentiated customer experience, or uncover new business opportunities. They do not understand what’s possible with respect to integrating Big Data into the business model.
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Taica manufacturers Alpha-GEL brand silicone components and materials, which maintain outstanding performance over a wide temperature range -40C to +200C. For more information, visit http://www.taica.co.jp/english/.
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, will discuss how they b...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
Digital transformation is changing the face of business. The IDC predicts that enterprises will commit to a massive new scale of digital transformation, to stake out leadership positions in the "digital transformation economy." Accordingly, attendees at the upcoming Cloud Expo | @ThingsExpo at the Santa Clara Convention Center in Santa Clara, CA, Oct 31-Nov 2, will find fresh new content in a new track called Enterprise Cloud & Digital Transformation.
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software. They hope to capture value from emerging technologies such as IoT, SDN, and AI. Ultimately, irrespective of the vertical, it is about deriving value from independent software applications participating in an ecosystem as one comprehensive solution. In his session at @ThingsExpo, Kausik Sridhar, founder and CTO of Pulzze Systems, will discuss how given the magnitude of today's applicati...
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...