Welcome!

Weblogic Authors: Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Handling System Core Files

Before you debug

This article will provide some useful tips for debugging your BEA WebLogic Server applications when a system core file occurs. It describes debugging tips, problem troubleshooting, and tools available to assist you in this process.

A system core file is usually indicative of an error in some native code. This could be from the application code of a user (if you are using native code [JNI] in your application), an error in the Java Virtual Machine version you are using, or in BEA WebLogic Server itself. There are a couple of places where native code could have caused the system core file to happen on your operating system. The following ideas and suggestions will help narrow the problem and stabilize the application until the exact cause of the system core can be determined.

The Java Virtual Machine
The first place to look is at the JVM itself. The JVM is a native program and can cause such errors. When in doubt, you can try another certified JVM and/or a later release to determine if a JVM bug is at fault. Many JVM bugs involve the use of the JIT compiler and disabling this feature will often resolve this type of problem. This can be done by supplying the "-Djava.compiler=none" option to the Java command line. Most of the time the JVM will produce a small log file that may contain useful information as to which library the system core may have come from; however, this is not true all of the time. The file is produced in the directory where BEA WebLogic Server was started and is of the form "hs_err_pid<PID #>.log", where "<PID #>" is the process ID of the BEA WebLogic Server process. On AIX this file would be "javacore<PID>.<ID Number>.txt", where "<PID #>" is the process ID of the BEA WebLogic Server process and "<ID Number>" is a number generated by the operating system. You can go to http://java.sun.com and search the Bug Database with the HotSpot "Error ID" number or the method that was happening when the error was reported.

Native Code
The next place to look is at any native libraries that are accessed with JNI (native code) calls in your application, if applicable. This can also cause a system core file to be produced. If your application uses such libraries, they should be carefully examined. It may be difficult to rule out these libraries, as their functionality may not be easily removed from the application. Extensive logging may be needed to determine if a pattern of use can be correlated with the system core dump/Dr. Watson error.

Another place is any Type 2 JDBC driver that makes use of native DBMS libraries, which could also produce this type of error. An option is to switch to a pure Java (Type 4) JDBC driver in order to determine if that is the cause. You can also check with the Type 2 JDBC driver provider to see if there are any known issues or if there is an updated version of the native libraries available.

A final place that could have caused this could be in the BEA WebLogic Server performance pack. This is also native code and when enabled could potentially produce such an error. One option is to disable this feature to determine if that is the cause. This can be done by adding "-Dweblogic.NativeIOEnabled=false" to the command line to start your server. If you are on an older service-pack version you can also check the list of fixed change requests for your particular BEA WebLogic Server version at http://edocs.bea.com.

System Core File
If after looking at the above items you cannot determine the cause of the error, then we can examine the system core file that was produced. If a system core file is produced it will be in the directory where BEA WebLogic Server was started. You will need to run an operating system-specific debugger on the resulting system core file to get a stacktrace that may help in pointing to the offending code.

If you are in development and the system core consistently happens, you can set the following flags to allow a thread dump to be taken of the server right before a core happens. This will get the state of the threads at that moment and may help narrow the problem to application code or point to a bug. The option is "-XX:+ShowMessageBoxOnError" on the Sun JVM (which is not officially documented on the Sun Web site). When the JVM crashes, the program will prompt: "Do you want to debug the problem?" You can then take a thread dump of the JVM. This option will be available on the 8.1 SP2 version of the BEA JRockit JVM when it becomes available. However, in that version the corresponding option will be "-Djrockit.waitonerror".

The best option is to run a debugger on the resulting system core file to get a stacktrace of the native calls. The following information was also given in my last article (WLDJ, Vol. 3, issue 2) but it is being repeated here for consistency. This information may help point out the offending code to you; or, if you are unsure, then contact BEA Customer Support with this information so they can investigate the stacktrace more thoroughly. If you are on a Windows platform, then a "Dr. Watson" file may be produced. Please send this file to BEA Customer Support when opening a case. Otherwise, check the following "Unix" operating system values to make sure that they have already been properly set in order to generate a core file:

  1. Check the "ulimit -c" (configured size of the core file) at a system and user level to make sure that it is set and that the value is not set too low to produce a meaningful core file.
  2. Check the available disk space for the user (for example, is there a disk quota?).
  3. Also check the following parameter, which on Solaris is usually in /etc/system file and can be used to disable core files: set sys:coredumpsize=0.
  4. On Linux, the core dump is turned off by default on all systems. In RedHat Advanced Server 2.1 it should be under "/etc/security". There should be a self-explanatory file called limits.conf; within that file look for the word "core". If it is set to "0", then coredump files are disabled.
  5. On HPUX check the setting called "kernel parm maxdsiz" (max_per_proc_data_size, which increases the User Process Data Segment Size) from the old value of, say, 64M to something higher like 134M.

Please get a stacktrace (or backtrace) from your debugger. Here are the commands needed when using "dbx" or "gdb", which will work on most platforms:

a. dbx:

  • $ java -version: Need to use right version of JDK
  • $ ls /opt/bin/dbx: Need to know dbx location, or "which dbx"
  • $ export: DEBUG_PROG=/opt/bin/dbx (or wherever "dbx" is located)
  • $ <path to java command>/java corefile

Now you will be in the debugger. Execute the following commands:

  • (dbx) where: Shows a summary of the stack
  • (dbx) threads: Shows the state of the existing threads
  • (dbx) quit

b. gdb

  • java -version: Need to use right version of jdk
  • ls /usr/local/bin/gdb: Need to know gdb location or "which gdb"
  • export DEBUG_PROG=/usr/local/bin/gdb: Or wherever "gdb" is located)
  • <path to java command>/java corefile

Now you will be in the debugger. Execute the following commands:

  • (gdb) where: Shows a summary of the stack
  • (gdb) thr: Switch among threads or show the current thread
  • (gdb) info thr: Inquire about existing threads
  • (gdb) thread apply 1 bt: Apply a command to a list of threads, specifically the backtrace to thread #1)
  • (gdb) quit

If you don't have access to a debugger you can check to see if you have the "pstack" and "pmap" utilities on your operating system. If you do have those utilities (on some operating systems you have to download these utilities separately); you can run them on the system core file to gather information for support. The syntax of the command would be something like this:
$ /usr/proc/bin/pstack core
$ /usr/proc/bin/pmap core

Further Information
You can also go to http://support.bea.com and find some published solutions on core files. In the "Question" field type in "S-16147" to display some additional information about using pstack and pmap on Solaris.

Enabling/Disabling Dr. Watson
The drwtsn32.log files are similar to system core files on Unix. On Windows 2000, these files are found in the following directory: C:\Documents and Settings\All Users\Documents\DrWatson. After entering "drwtsn32 ?" in a DOS command window and hitting enter, the "Dr. Watson for Windows 2000" box appears. The DrWatson log file overview option will display a screen that explains the format of the drwtsn32.log files.

By default, Dr. Watson will be enabled when Windows NT is installed. To disable Dr. Watson, the following Registry value must be changed from 1 to 0 (zero): "\HKEY_LOCAL_MACHINE\SOFTWARE \Microsoft\Windows NT\CurrentVersion\AeDebug". An entry called "Auto" corresponds to how Dr. Watson will start up. To enable Dr. Watson, change the "Auto" value from 0 (zero) to 1. This will then launch whatever debugger, or application, is under the Debugger registry value. For Dr. Watson, the Debugger value should contain:

drwtsn32 -p %ld -e %ld -g

Links to On-Line Debugger Manuals

  1. gdb: www.gnu.org/manual/gdb-5.1.1/gdb.html
  2. dbx:
    a. Sun: http://docs.sun.com/db/doc/805-4948?q=DBX
    b. IBM: http://publib16.boulder.ibm.com/pseries/en_US/cmds/aixcmds2/dbx.htm
  3. adb: HP: http://docs.hp.com/hpux/onlinedocs/B2355-90680/00/00/8-con.html

If none of these hints help direct you towards a solution or an identifier in your application, then you should contact BEA Customer Support for further diagnosis. You can open a case with a valid support contract by logging in at http://support.bea.com/login.jsp.

More Stories By Steve Pozarycki

Steven Pozarycki is a backline developer relations engineer with BEA Systems, Customer Support Group. He specializes in troubleshooting and solving complex customer issues with their mission-critical applications on BEA products. Steve holds a bachelor's degree in computer science.

Reproduced with permission from BEA Systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Bsquare has been named “Silver Sponsor” of SYS-CON's @ThingsExpo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. For more than two decades, Bsquare has helped its customers extract business value from a broad array of physical assets by making them intelligent, connecting them, and using the data they generate to optimize business processes.
I'm a lonely sensor. I spend all day telling the world how I'm feeling, but none of the other sensors seem to care. I want to be connected. I want to build relationships with other sensors to be more useful for my human. I want my human to understand that when my friends next door are too hot for a while, I'll soon be flaming. And when all my friends go outside without me, I may be left behind. Don't just log my data; use the relationship graph. In his session at @ThingsExpo, Ryan Boyd, Engi...
SYS-CON Events announced today that Pulzze Systems will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Pulzze Systems, Inc. provides infrastructure products for the Internet of Things to enable any connected device and system to carry out matched operations without programming. For more information, visit http://www.pulzzesystems.com.
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management solutions, helping companies worldwide activate their data to drive more value and business insight and to transform moder...
Fact is, enterprises have significant legacy voice infrastructure that’s costly to replace with pure IP solutions. How can we bring this analog infrastructure into our shiny new cloud applications? There are proven methods to bind both legacy voice applications and traditional PSTN audio into cloud-based applications and services at a carrier scale. Some of the most successful implementations leverage WebRTC, WebSockets, SIP and other open source technologies. In his session at @ThingsExpo, Da...
Almost two-thirds of companies either have or soon will have IoT as the backbone of their business in 2016. However, IoT is far more complex than most firms expected. How can you not get trapped in the pitfalls? In his session at @ThingsExpo, Tony Shan, a renowned visionary and thought leader, will introduce a holistic method of IoTification, which is the process of IoTifying the existing technology and business models to adopt and leverage IoT. He will drill down to the components in this fra...
The vision of a connected smart home is becoming reality with the application of integrated wireless technologies in devices and appliances. The use of standardized and TCP/IP networked wireless technologies in line-powered and battery operated sensors and controls has led to the adoption of radios in the 2.4GHz band, including Wi-Fi, BT/BLE and 802.15.4 applied ZigBee and Thread. This is driving the need for robust wireless coexistence for multiple radios to ensure throughput performance and th...
Enterprise IT has been in the era of Hybrid Cloud for some time now. But it seems most conversations about Hybrid are focused on integrating AWS, Microsoft Azure, or Google ECM into existing on-premises systems. Where is all the Private Cloud? What do technology providers need to do to make their offerings more compelling? How should enterprise IT executives and buyers define their focus, needs, and roadmap, and communicate that clearly to the providers?
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
There is little doubt that Big Data solutions will have an increasing role in the Enterprise IT mainstream over time. Big Data at Cloud Expo - to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA - has announced its Call for Papers is open. Cloud computing is being adopted in one form or another by 94% of enterprises today. Tens of billions of new devices are being connected to The Internet of Things. And Big Data is driving this bus. An exponential increase is...
Digital innovation is the next big wave of business transformation based on digital technologies of which IoT and Big Data are key components, For example: Business boundary innovation is a challenge to excavate third-party business value using IoT and BigData, like Nest Business structure innovation may propose re-building business structure from scratch, as Uber does in the taxicab industry The social model innovation is also a big challenge to the new social architecture with the design fr...
DevOps at Cloud Expo, taking place Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long dev...
Data is an unusual currency; it is not restricted by the same transactional limitations as money or people. In fact, the more that you leverage your data across multiple business use cases, the more valuable it becomes to the organization. And the same can be said about the organization’s analytics. In his session at 19th Cloud Expo, Bill Schmarzo, CTO for the Big Data Practice at EMC, will introduce a methodology for capturing, enriching and sharing data (and analytics) across the organizati...
IoT is fundamentally transforming the auto industry, turning the vehicle into a hub for connected services, including safety, infotainment and usage-based insurance. Auto manufacturers – and businesses across all verticals – have built an entire ecosystem around the Connected Car, creating new customer touch points and revenue streams. In his session at @ThingsExpo, Macario Namie, Head of IoT Strategy at Cisco Jasper, will share real-world examples of how IoT transforms the car from a static p...
The many IoT deployments around the world are busy integrating smart devices and sensors into their enterprise IT infrastructures. Yet all of this technology – and there are an amazing number of choices – is of no use without the software to gather, communicate, and analyze the new data flows. Without software, there is no IT. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the protocols that communicate data and the emerging data analy...
SYS-CON Events announced today that China Unicom will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. China United Network Communications Group Co. Ltd ("China Unicom") was officially established in 2009 on the basis of the merger of former China Netcom and former China Unicom. China Unicom mainly operates a full range of telecommunications services including mobile broadband (GSM, WCDMA, LTE F...
The Transparent Cloud-computing Consortium (abbreviation: T-Cloud Consortium) will conduct research activities into changes in the computing model as a result of collaboration between "device" and "cloud" and the creation of new value and markets through organic data processing High speed and high quality networks, and dramatic improvements in computer processing capabilities, have greatly changed the nature of applications and made the storing and processing of data on the network commonplace.
Video experiences should be unique and exciting! But that doesn’t mean you need to patch all the pieces yourself. Users demand rich and engaging experiences and new ways to connect with you. But creating robust video applications at scale can be complicated, time-consuming and expensive. In his session at @ThingsExpo, Zohar Babin, Vice President of Platform, Ecosystem and Community at Kaltura, will discuss how VPaaS enables you to move fast, creating scalable video experiences that reach your...
Big Data has been changing the world. IoT fuels the further transformation recently. How are Big Data and IoT related? In his session at @BigDataExpo, Tony Shan, a renowned visionary and thought leader, will explore the interplay of Big Data and IoT. He will anatomize Big Data and IoT separately in terms of what, which, why, where, when, who, how and how much. He will then analyze the relationship between IoT and Big Data, specifically the drilldown of how the 4Vs of Big Data (Volume, Variety,...
If you’re responsible for an application that depends on the data or functionality of various IoT endpoints – either sensors or devices – your brand reputation depends on the security, reliability, and compliance of its many integrated parts. If your application fails to deliver the expected business results, your customers and partners won't care if that failure stems from the code you developed or from a component that you integrated. What can you do to ensure that the endpoints work as expect...