Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Java IoT, Weblogic

Java IoT: Article

Java Thread Dumps

Analyzing Java Thread Dumps

Software maintenance is a tedious and challenging job. As long as the software functions as expected, it looks great. Imagine the situation, your pager keeps buzzing in the midnight (Not a happy feeling, right?).

Any software system, no matter how well it has been built and quality tested might get into run-time performance issues. The reasons can be within the functional boundary or can be from the external environment. Software systems are built on certain assumptions and pre-conceived notions. However, when they go live, the assumptions may turn out to be false causing the system to malfunction.

In enterprise J2EE systems which generally have very large user base and involve interactions between heterogeneous systems, one of the general runtime issues reported is the system slowdown or system "hang". In such situations, the general trouble shooting pattern would be to analyze the Java thread dumps to isolate the threads which are causing the slow-down or hang. This article discusses the Java stack traces; anatomy of Java Threads and how to read thread dumps in general.

Exceptions and Stack Traces
All of us encounter and have encountered exceptions during our learning/development phases. Exception is the way Java reports a runtime error. Exceptions have two parts. Message and the stack trace. Message tells you what has gone wrong. Stack Traces provide a complete call flow through all the classes involved top-down as part of the runtime error.

The following is an example stack trace for an ArrayIndexOutOfBoundsException

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4

at Test.run(Test.java:13)

at Test.<init>(Test.java:5)

at Test.main(Test.java:20)

In the above exception, the first line "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4" tells you that the JVM encountered the reported exception when it tried to access the value at array index 4. The java thread which encountered the exception is "main" thread.

Next let's read through the stack trace. The rule of thumb for exceptions is to read from the top line (message line) to understand what the exception is and keep going down to understand the call flow. In the above example, the call flow started at Test.java line number 20 (main() method) and then it called the constructor of the Test class. Constructors are represented with <init> in the stack traces. Then it moved on to the run() method in the Test class and encountered the reported exception at line number 13.

From the above stack trace, we can conclude that in Test.java, an attempt is made to read past the size of the Array.

The Java Thread Dump
Java Thread dump can be defined as a snapshot of all threads running inside a Java Virtual Machine (JVM) at a given point of time. A thread dump might contain a single thread or multiple threads. In multi threaded environments like J2EE application servers, there will be many threads and Thread Groups. Each of these threads will have its own call stack and will be performing a separate function at a given time. Thread Dump will provide the stack traces of all the JVM threads and much more information about a particular thread.

The Java VM Process and Java Threads
The Java Virtual Machine or JVM is an operating system level process. Java Threads are child processes or Light-Weight Processes (in Solaris terms) of the JVM process.

Generating Java Thread Dumps
Thread Dump is generated by sending a SIGQUIT signal to the JVM process. There are different ways of sending this signal to the process.

In Unix, use "kill -3 <pid>" where pid is the Process ID of the JVM.

In Windows, press CTRL+BREAK on the window where the JVM is running

Java Thread State
Each Java Thread can be in one of the four states during its lifetime.

Runnable - The thread is either running or ready to run when it gets its CPU turn. JRockit thread dumps refer to this state as ACTIVE.

Waiting on Monitor - The thread is either sleeping or waiting on an object for said period of time or waiting to be notified by another thread. This will happen when any of the sleep() methods on Thread object or wait() method on an Object is called.

For example, in WebLogc server the idle execute threads are in this condition and they wait till a socket reader thread notify them of some new work to be done. The stack trace will look like this:

"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x1752F040 nid=0x180c in Object.wait() [1887f000..1887fd8c]

at java.lang.Object.wait(Native Method) waiting on <04134D98> (a weblogic.kernel.ExecuteThread)

at java.lang.Object.wait(Object.java:426)

at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:126)

locked <04134D98> (a weblogic.kernel.ExecuteThread)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:145)

Some other versions of the JVM call this state as CW, Object.wait() (as above). JRockit refer to this as state as WAITING.

Waiting for Monitor Entry - The thread is waiting to get the lock for an object (some other thread may be holding the lock). This happens if two or more threads try to execute synchronized code. Note that the lock is always for an object and not for individual methods.

Sample stack trace of a thread in this condition:

"ExecuteThread: '24' for queue: 'DisplayExecuteQueue'" daemon prio=5 tid=0x5541b0 nid=0x3b waiting for monitor entry [49b7f000..49b7fc24]

at weblogic.cluster.replication.ReplicationManager.createSecondary (ReplicationManager.java:908)
- waiting to lock <6c4b9130> (a java.lang.Object)
at weblogic.cluster.replication.ReplicationManager.updateSecondary (ReplicationManager.java:715)
at weblogic.servlet.internal.session.ReplicatedSessionData.syncSession (ReplicatedSessonData.java:459)
- locked <6c408700> (a weblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.session.ReplicatedSessionContext.sync (ReplicatedSessionContext.java:134)
- locked <6c408700> (aweblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.ServletRequestImpl.syncSession (ServletRequestImpl.java:2418)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:3137)
at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2544)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)

In the above stacktrace, you can see that this thread holds the lock an object (6c408700) and waiting to lock another object (6c4b9130)

Some other JVMs may not give the object Ids with lock information in the stack trace. The same state may also be called as ‘MW'. JRockit refers to this state as LOCKED.

Anatomy of a Java Thread
To be able to read/analyze thread dump, it is important to understand various parts of a Thread dump. Let's take a simple Thread Stack example and read various parts of it.

"ExecuteThread: '1' " daemon prio=5 tid=0x628330 nid=0xf runnable [0xe4881000..0xe48819e0]

at com.vantive.vanjavi.VanJavi.VanCreateForm(Native Method)

at com.vantive.vanjavi.VanMain.open(VanMain.java:53)

at jsp_servlet._so.__newServiceOrder.printSOSection( __newServiceOrder.java:3547)

at jsp_servlet._so.__newServiceOrder._jspService (__newServiceOrder.java:5652)

at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:265)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:200)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:2495)

at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2204)

at weblogic.kernel.ExecuteThread.execute (ExecuteThread.java:139)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

In the above Thread Dump, the interesting part to is the first line. The rest of the stuff is nothing more than a general stack trace. Lets analyze the first line here

Execute Thread : 1 This indicates the name of the thread.

daemon Indicates that this Thread is a daemon thread

prio=5 The priority of this thread (default is 5)

tid The Java Thread Id (Unique identifier for this thread in the running JVM instance).

nid Native Identifier of the thread. Indicates LWP id in Solaris. Identifier for this process

at the OS level.

runnable Indicate the Thread state (See above)

[x..y] Indicates the range of addresses in the heap where this thread is executing.

The remaining part of the thread dump show the call flow. In this example, the thread (Execute Thread 1) is a OS daemon which is currently executing a native method vanCreateForm().

Putting Thread Dumps to Use
In this section I describe few use cases in which Thread dumps are very useful.

High CPU consumption
Diagnosis

The application seems to consume almost 100% CPU and the throughput has come down drastically. Starts to perform very poorly with high load on the CPU.

Thread Dump
All threads in the dump generally show one or more threads struck doing the same operation through all the thread dumps.

Solution

  • Take a series of thread dumps (around 5-7) for one particular call flow (a web form submit for example) before the call is completed.
  • Looking for "runnable" threads in the Thread Dump. If each such thread seems to move (the method call in each thread dump is different), then the thread is progressing and probably not a culprit. If the thread is executing the same method (same line number) through all the thread dumps, then it is almost for sure the culprit. Go to the code and do the code level analysis. You are almost certain to catch an issue there.

Low CPU consumption and Poor Response time
Diagnosis

This happens in a high I/O bound systems under high load. The CPU consumption is low with only few threads consuming reasonable CPU chunk. However the application response time will be very slow.

Thread Dump
Some or all runnable threads seem to struck performing an I/O operation like a File read/write or a database operation.

Solution
Understand the I/O operation profile of your application. Use caches where applicable to reduce DB interactions.

Application/Server Hang
Diagnosis

An application or an server JVM hosting the application will hang (become unresponsive).

Thread Dump

  • All runnable threads are struck doing the same operation through all the thread dumps taken. The server runs out of any more threads as all the runnable threads "never" complete their operation.
  • There might be many threads waiting for monitor entry. This can happen if a "runnable" thread is holding a lock on an object and never returns it while others are waiting for the same lock.

Solution

  • Check for dead lock. JVM generally detects a deadlock if it is simple enough (Thread A waiting for Thread B and vice versa). However, you need to understand the lock profile at the given moment to see if there is any complex deadlock situation involved.
  • Revisit the synchronized methods/block in the code. Reduce the size of synchronization area to the extent possible.
  • One issue could be too long a timeout while accessing a remote resource/component. Implement a reasonable timeout on the remote object clients so that the throw appropriate exception when a remote system does not respond in a reasonable amount of time.
  • If all the threads are waiting for a resource (like an EJB/DB connection), consider increasing the object pool size for those resources.

Tools
There are both commercial and open source tools available for thread dump analysis. One such tool is Samurai. It is a light weight open source tool which runs as a Java Web Start application as well as from your command prompt. For more information and documentation on Samurai, visit

http://yusuke.homeip.net/samurai/en/index.html

Conclusion
Maintaining J2EE enterprise application in a production environment is a tough job. As the business dynamics change, J2EE application environments change which might cause runtime instability in the production application. One of the primary factors that effect an running J2EE application is high load. While most systems are designed to be scalable, environmental limitations might cause them to become non-responsive.

Java Thread Dumps is an excellent mechanism to identify, diagnose, detect and resolve typical production issues. While application profiling and other mechanisms do exist, analyzing Java Thread dumps will provide us an clear and early understanding of popular production level issues there by saving time and helping us provide better user experience with the production applications.

More Stories By Shankar Itchapurapu

Shankar Itchapurapu is a software engineer at Oracle in India. He holds a Master's degree in Computer Applications. You can e-mail Shankar at [email protected]

@ThingsExpo Stories
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
SYS-CON Events announced today that Enroute Lab will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enroute Lab is an industrial design, research and development company of unmanned robotic vehicle system. For more information, please visit http://elab.co.jp/.
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capabi...
SYS-CON Events announced today that Suzuki Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Suzuki Inc. is a semiconductor-related business, including sales of consuming parts, parts repair, and maintenance for semiconductor manufacturing machines, etc. It is also a health care business providing experimental research for...
Real IoT production deployments running at scale are collecting sensor data from hundreds / thousands / millions of devices. The goal is to take business-critical actions on the real-time data and find insights from stored datasets. In his session at @ThingsExpo, John Walicki, Watson IoT Developer Advocate at IBM Cloud, will provide a fast-paced developer journey that follows the IoT sensor data from generation, to edge gateway, to edge analytics, to encryption, to the IBM Bluemix cloud, to Wa...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
SYS-CON Events announced today that Nihon Micron will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nihon Micron Co., Ltd. strives for technological innovation to establish high-density, high-precision processing technology for providing printed circuit board and metal mount RFID tags used for communication devices. For more inf...
SYS-CON Events announced today that mruby Forum will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. mruby is the lightweight implementation of the Ruby language. We introduce mruby and the mruby IoT framework that enhances development productivity. For more information, visit http://forum.mruby.org/.
In his session at @ThingsExpo, Greg Gorman is the Director, IoT Developer Ecosystem, Watson IoT, will provide a short tutorial on Node-RED, a Node.js-based programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using a wide range of nodes in the palette that can be deployed to its runtime in a single-click. There is a large library of contributed nodes that help so...
SYS-CON Events announced today that Ryobi Systems will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ryobi Systems Co., Ltd., as an information service company, specialized in business support for local governments and medical industry. We are challenging to achive the precision farming with AI. For more information, visit http:...
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
SYS-CON Events announced today that Daiya Industry will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Daiya Industry specializes in orthotic support systems and assistive devices with pneumatic artificial muscles in order to contribute to an extended healthy life expectancy. For more information, please visit https://www.daiyak...
SYS-CON Events announced today that B2Cloud will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. B2Cloud specializes in IoT devices for preventive and predictive maintenance in any kind of equipment retrieving data like Energy consumption, working time, temperature, humidity, pressure, etc.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp em...
What is the best strategy for selecting the right offshore company for your business? In his session at 21st Cloud Expo, Alan Winters, U.S. Head of Business Development at MobiDev, will discuss the things to look for - positive and negative - in evaluating your options. He will also discuss how to maximize productivity with your offshore developers. Before you start your search, clearly understand your business needs and how that impacts software choices.
SYS-CON Events announced today that Interface Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Interface Corporation is a company developing, manufacturing and marketing high quality and wide variety of industrial computers and interface modules such as PCIs and PCI express. For more information, visit http://www.i...
SYS-CON Events announced today that Keisoku Research Consultant Co. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Keisoku Research Consultant, Co. offers research and consulting in a wide range of civil engineering-related fields from information construction to preservation of cultural properties. For more information, vi...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...