Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Java IoT, Weblogic

Java IoT: Article

Java Thread Dumps

Analyzing Java Thread Dumps

Software maintenance is a tedious and challenging job. As long as the software functions as expected, it looks great. Imagine the situation, your pager keeps buzzing in the midnight (Not a happy feeling, right?).

Any software system, no matter how well it has been built and quality tested might get into run-time performance issues. The reasons can be within the functional boundary or can be from the external environment. Software systems are built on certain assumptions and pre-conceived notions. However, when they go live, the assumptions may turn out to be false causing the system to malfunction.

In enterprise J2EE systems which generally have very large user base and involve interactions between heterogeneous systems, one of the general runtime issues reported is the system slowdown or system "hang". In such situations, the general trouble shooting pattern would be to analyze the Java thread dumps to isolate the threads which are causing the slow-down or hang. This article discusses the Java stack traces; anatomy of Java Threads and how to read thread dumps in general.

Exceptions and Stack Traces
All of us encounter and have encountered exceptions during our learning/development phases. Exception is the way Java reports a runtime error. Exceptions have two parts. Message and the stack trace. Message tells you what has gone wrong. Stack Traces provide a complete call flow through all the classes involved top-down as part of the runtime error.

The following is an example stack trace for an ArrayIndexOutOfBoundsException

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4

at Test.run(Test.java:13)

at Test.<init>(Test.java:5)

at Test.main(Test.java:20)

In the above exception, the first line "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4" tells you that the JVM encountered the reported exception when it tried to access the value at array index 4. The java thread which encountered the exception is "main" thread.

Next let's read through the stack trace. The rule of thumb for exceptions is to read from the top line (message line) to understand what the exception is and keep going down to understand the call flow. In the above example, the call flow started at Test.java line number 20 (main() method) and then it called the constructor of the Test class. Constructors are represented with <init> in the stack traces. Then it moved on to the run() method in the Test class and encountered the reported exception at line number 13.

From the above stack trace, we can conclude that in Test.java, an attempt is made to read past the size of the Array.

The Java Thread Dump
Java Thread dump can be defined as a snapshot of all threads running inside a Java Virtual Machine (JVM) at a given point of time. A thread dump might contain a single thread or multiple threads. In multi threaded environments like J2EE application servers, there will be many threads and Thread Groups. Each of these threads will have its own call stack and will be performing a separate function at a given time. Thread Dump will provide the stack traces of all the JVM threads and much more information about a particular thread.

The Java VM Process and Java Threads
The Java Virtual Machine or JVM is an operating system level process. Java Threads are child processes or Light-Weight Processes (in Solaris terms) of the JVM process.

Generating Java Thread Dumps
Thread Dump is generated by sending a SIGQUIT signal to the JVM process. There are different ways of sending this signal to the process.

In Unix, use "kill -3 <pid>" where pid is the Process ID of the JVM.

In Windows, press CTRL+BREAK on the window where the JVM is running

Java Thread State
Each Java Thread can be in one of the four states during its lifetime.

Runnable - The thread is either running or ready to run when it gets its CPU turn. JRockit thread dumps refer to this state as ACTIVE.

Waiting on Monitor - The thread is either sleeping or waiting on an object for said period of time or waiting to be notified by another thread. This will happen when any of the sleep() methods on Thread object or wait() method on an Object is called.

For example, in WebLogc server the idle execute threads are in this condition and they wait till a socket reader thread notify them of some new work to be done. The stack trace will look like this:

"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x1752F040 nid=0x180c in Object.wait() [1887f000..1887fd8c]

at java.lang.Object.wait(Native Method) waiting on <04134D98> (a weblogic.kernel.ExecuteThread)

at java.lang.Object.wait(Object.java:426)

at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:126)

locked <04134D98> (a weblogic.kernel.ExecuteThread)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:145)

Some other versions of the JVM call this state as CW, Object.wait() (as above). JRockit refer to this as state as WAITING.

Waiting for Monitor Entry - The thread is waiting to get the lock for an object (some other thread may be holding the lock). This happens if two or more threads try to execute synchronized code. Note that the lock is always for an object and not for individual methods.

Sample stack trace of a thread in this condition:

"ExecuteThread: '24' for queue: 'DisplayExecuteQueue'" daemon prio=5 tid=0x5541b0 nid=0x3b waiting for monitor entry [49b7f000..49b7fc24]

at weblogic.cluster.replication.ReplicationManager.createSecondary (ReplicationManager.java:908)
- waiting to lock <6c4b9130> (a java.lang.Object)
at weblogic.cluster.replication.ReplicationManager.updateSecondary (ReplicationManager.java:715)
at weblogic.servlet.internal.session.ReplicatedSessionData.syncSession (ReplicatedSessonData.java:459)
- locked <6c408700> (a weblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.session.ReplicatedSessionContext.sync (ReplicatedSessionContext.java:134)
- locked <6c408700> (aweblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.ServletRequestImpl.syncSession (ServletRequestImpl.java:2418)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:3137)
at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2544)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)

In the above stacktrace, you can see that this thread holds the lock an object (6c408700) and waiting to lock another object (6c4b9130)

Some other JVMs may not give the object Ids with lock information in the stack trace. The same state may also be called as ‘MW'. JRockit refers to this state as LOCKED.

Anatomy of a Java Thread
To be able to read/analyze thread dump, it is important to understand various parts of a Thread dump. Let's take a simple Thread Stack example and read various parts of it.

"ExecuteThread: '1' " daemon prio=5 tid=0x628330 nid=0xf runnable [0xe4881000..0xe48819e0]

at com.vantive.vanjavi.VanJavi.VanCreateForm(Native Method)

at com.vantive.vanjavi.VanMain.open(VanMain.java:53)

at jsp_servlet._so.__newServiceOrder.printSOSection( __newServiceOrder.java:3547)

at jsp_servlet._so.__newServiceOrder._jspService (__newServiceOrder.java:5652)

at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:265)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:200)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:2495)

at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2204)

at weblogic.kernel.ExecuteThread.execute (ExecuteThread.java:139)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

In the above Thread Dump, the interesting part to is the first line. The rest of the stuff is nothing more than a general stack trace. Lets analyze the first line here

Execute Thread : 1 This indicates the name of the thread.

daemon Indicates that this Thread is a daemon thread

prio=5 The priority of this thread (default is 5)

tid The Java Thread Id (Unique identifier for this thread in the running JVM instance).

nid Native Identifier of the thread. Indicates LWP id in Solaris. Identifier for this process

at the OS level.

runnable Indicate the Thread state (See above)

[x..y] Indicates the range of addresses in the heap where this thread is executing.

The remaining part of the thread dump show the call flow. In this example, the thread (Execute Thread 1) is a OS daemon which is currently executing a native method vanCreateForm().

Putting Thread Dumps to Use
In this section I describe few use cases in which Thread dumps are very useful.

High CPU consumption
Diagnosis

The application seems to consume almost 100% CPU and the throughput has come down drastically. Starts to perform very poorly with high load on the CPU.

Thread Dump
All threads in the dump generally show one or more threads struck doing the same operation through all the thread dumps.

Solution

  • Take a series of thread dumps (around 5-7) for one particular call flow (a web form submit for example) before the call is completed.
  • Looking for "runnable" threads in the Thread Dump. If each such thread seems to move (the method call in each thread dump is different), then the thread is progressing and probably not a culprit. If the thread is executing the same method (same line number) through all the thread dumps, then it is almost for sure the culprit. Go to the code and do the code level analysis. You are almost certain to catch an issue there.

Low CPU consumption and Poor Response time
Diagnosis

This happens in a high I/O bound systems under high load. The CPU consumption is low with only few threads consuming reasonable CPU chunk. However the application response time will be very slow.

Thread Dump
Some or all runnable threads seem to struck performing an I/O operation like a File read/write or a database operation.

Solution
Understand the I/O operation profile of your application. Use caches where applicable to reduce DB interactions.

Application/Server Hang
Diagnosis

An application or an server JVM hosting the application will hang (become unresponsive).

Thread Dump

  • All runnable threads are struck doing the same operation through all the thread dumps taken. The server runs out of any more threads as all the runnable threads "never" complete their operation.
  • There might be many threads waiting for monitor entry. This can happen if a "runnable" thread is holding a lock on an object and never returns it while others are waiting for the same lock.

Solution

  • Check for dead lock. JVM generally detects a deadlock if it is simple enough (Thread A waiting for Thread B and vice versa). However, you need to understand the lock profile at the given moment to see if there is any complex deadlock situation involved.
  • Revisit the synchronized methods/block in the code. Reduce the size of synchronization area to the extent possible.
  • One issue could be too long a timeout while accessing a remote resource/component. Implement a reasonable timeout on the remote object clients so that the throw appropriate exception when a remote system does not respond in a reasonable amount of time.
  • If all the threads are waiting for a resource (like an EJB/DB connection), consider increasing the object pool size for those resources.

Tools
There are both commercial and open source tools available for thread dump analysis. One such tool is Samurai. It is a light weight open source tool which runs as a Java Web Start application as well as from your command prompt. For more information and documentation on Samurai, visit

http://yusuke.homeip.net/samurai/en/index.html

Conclusion
Maintaining J2EE enterprise application in a production environment is a tough job. As the business dynamics change, J2EE application environments change which might cause runtime instability in the production application. One of the primary factors that effect an running J2EE application is high load. While most systems are designed to be scalable, environmental limitations might cause them to become non-responsive.

Java Thread Dumps is an excellent mechanism to identify, diagnose, detect and resolve typical production issues. While application profiling and other mechanisms do exist, analyzing Java Thread dumps will provide us an clear and early understanding of popular production level issues there by saving time and helping us provide better user experience with the production applications.

More Stories By Shankar Itchapurapu

Shankar Itchapurapu is a software engineer at Oracle in India. He holds a Master's degree in Computer Applications. You can e-mail Shankar at [email protected]

@ThingsExpo Stories
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
Detecting internal user threats in the Big Data eco-system is challenging and cumbersome. Many organizations monitor internal usage of the Big Data eco-system using a set of alerts. This is not a scalable process given the increase in the number of alerts with the accelerating growth in data volume and user base. Organizations are increasingly leveraging machine learning to monitor only those data elements that are sensitive and critical, autonomously establish monitoring policies, and to detect...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
In his session at @ThingsExpo, Dr. Robert Cohen, an economist and senior fellow at the Economic Strategy Institute, presented the findings of a series of six detailed case studies of how large corporations are implementing IoT. The session explored how IoT has improved their economic performance, had major impacts on business models and resulted in impressive ROIs. The companies covered span manufacturing and services firms. He also explored servicification, how manufacturing firms shift from se...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
IoT solutions exploit operational data generated by Internet-connected smart “things” for the purpose of gaining operational insight and producing “better outcomes” (for example, create new business models, eliminate unscheduled maintenance, etc.). The explosive proliferation of IoT solutions will result in an exponential growth in the volume of IoT data, precipitating significant Information Governance issues: who owns the IoT data, what are the rights/duties of IoT solutions adopters towards t...
Amazon started as an online bookseller 20 years ago. Since then, it has evolved into a technology juggernaut that has disrupted multiple markets and industries and touches many aspects of our lives. It is a relentless technology and business model innovator driving disruption throughout numerous ecosystems. Amazon’s AWS revenues alone are approaching $16B a year making it one of the largest IT companies in the world. With dominant offerings in Cloud, IoT, eCommerce, Big Data, AI, Digital Assista...
Organizations planning enterprise data center consolidation and modernization projects are faced with a challenging, costly reality. Requirements to deploy modern, cloud-native applications simultaneously with traditional client/server applications are almost impossible to achieve with hardware-centric enterprise infrastructure. Compute and network infrastructure are fast moving down a software-defined path, but storage has been a laggard. Until now.
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
IoT is at the core or many Digital Transformation initiatives with the goal of re-inventing a company's business model. We all agree that collecting relevant IoT data will result in massive amounts of data needing to be stored. However, with the rapid development of IoT devices and ongoing business model transformation, we are not able to predict the volume and growth of IoT data. And with the lack of IoT history, traditional methods of IT and infrastructure planning based on the past do not app...
DXWorldEXPO LLC announced today that the upcoming DXWorldEXPO | CloudEXPO New York event will feature 10 companies from Poland to participate at the "Poland Digital Transformation Pavilion" on November 12-13, 2018.