Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Handling System Core Files

Before you debug

This article will provide some useful tips for debugging your BEA WebLogic Server applications when a system core file occurs. It describes debugging tips, problem troubleshooting, and tools available to assist you in this process.

A system core file is usually indicative of an error in some native code. This could be from the application code of a user (if you are using native code [JNI] in your application), an error in the Java Virtual Machine version you are using, or in BEA WebLogic Server itself. There are a couple of places where native code could have caused the system core file to happen on your operating system. The following ideas and suggestions will help narrow the problem and stabilize the application until the exact cause of the system core can be determined.

The Java Virtual Machine
The first place to look is at the JVM itself. The JVM is a native program and can cause such errors. When in doubt, you can try another certified JVM and/or a later release to determine if a JVM bug is at fault. Many JVM bugs involve the use of the JIT compiler and disabling this feature will often resolve this type of problem. This can be done by supplying the "-Djava.compiler=none" option to the Java command line. Most of the time the JVM will produce a small log file that may contain useful information as to which library the system core may have come from; however, this is not true all of the time. The file is produced in the directory where BEA WebLogic Server was started and is of the form "hs_err_pid<PID #>.log", where "<PID #>" is the process ID of the BEA WebLogic Server process. On AIX this file would be "javacore<PID>.<ID Number>.txt", where "<PID #>" is the process ID of the BEA WebLogic Server process and "<ID Number>" is a number generated by the operating system. You can go to http://java.sun.com and search the Bug Database with the HotSpot "Error ID" number or the method that was happening when the error was reported.

Native Code
The next place to look is at any native libraries that are accessed with JNI (native code) calls in your application, if applicable. This can also cause a system core file to be produced. If your application uses such libraries, they should be carefully examined. It may be difficult to rule out these libraries, as their functionality may not be easily removed from the application. Extensive logging may be needed to determine if a pattern of use can be correlated with the system core dump/Dr. Watson error.

Another place is any Type 2 JDBC driver that makes use of native DBMS libraries, which could also produce this type of error. An option is to switch to a pure Java (Type 4) JDBC driver in order to determine if that is the cause. You can also check with the Type 2 JDBC driver provider to see if there are any known issues or if there is an updated version of the native libraries available.

A final place that could have caused this could be in the BEA WebLogic Server performance pack. This is also native code and when enabled could potentially produce such an error. One option is to disable this feature to determine if that is the cause. This can be done by adding "-Dweblogic.NativeIOEnabled=false" to the command line to start your server. If you are on an older service-pack version you can also check the list of fixed change requests for your particular BEA WebLogic Server version at http://edocs.bea.com.

System Core File
If after looking at the above items you cannot determine the cause of the error, then we can examine the system core file that was produced. If a system core file is produced it will be in the directory where BEA WebLogic Server was started. You will need to run an operating system-specific debugger on the resulting system core file to get a stacktrace that may help in pointing to the offending code.

If you are in development and the system core consistently happens, you can set the following flags to allow a thread dump to be taken of the server right before a core happens. This will get the state of the threads at that moment and may help narrow the problem to application code or point to a bug. The option is "-XX:+ShowMessageBoxOnError" on the Sun JVM (which is not officially documented on the Sun Web site). When the JVM crashes, the program will prompt: "Do you want to debug the problem?" You can then take a thread dump of the JVM. This option will be available on the 8.1 SP2 version of the BEA JRockit JVM when it becomes available. However, in that version the corresponding option will be "-Djrockit.waitonerror".

The best option is to run a debugger on the resulting system core file to get a stacktrace of the native calls. The following information was also given in my last article (WLDJ, Vol. 3, issue 2) but it is being repeated here for consistency. This information may help point out the offending code to you; or, if you are unsure, then contact BEA Customer Support with this information so they can investigate the stacktrace more thoroughly. If you are on a Windows platform, then a "Dr. Watson" file may be produced. Please send this file to BEA Customer Support when opening a case. Otherwise, check the following "Unix" operating system values to make sure that they have already been properly set in order to generate a core file:

  1. Check the "ulimit -c" (configured size of the core file) at a system and user level to make sure that it is set and that the value is not set too low to produce a meaningful core file.
  2. Check the available disk space for the user (for example, is there a disk quota?).
  3. Also check the following parameter, which on Solaris is usually in /etc/system file and can be used to disable core files: set sys:coredumpsize=0.
  4. On Linux, the core dump is turned off by default on all systems. In RedHat Advanced Server 2.1 it should be under "/etc/security". There should be a self-explanatory file called limits.conf; within that file look for the word "core". If it is set to "0", then coredump files are disabled.
  5. On HPUX check the setting called "kernel parm maxdsiz" (max_per_proc_data_size, which increases the User Process Data Segment Size) from the old value of, say, 64M to something higher like 134M.

Please get a stacktrace (or backtrace) from your debugger. Here are the commands needed when using "dbx" or "gdb", which will work on most platforms:

a. dbx:

  • $ java -version: Need to use right version of JDK
  • $ ls /opt/bin/dbx: Need to know dbx location, or "which dbx"
  • $ export: DEBUG_PROG=/opt/bin/dbx (or wherever "dbx" is located)
  • $ <path to java command>/java corefile

Now you will be in the debugger. Execute the following commands:

  • (dbx) where: Shows a summary of the stack
  • (dbx) threads: Shows the state of the existing threads
  • (dbx) quit

b. gdb

  • java -version: Need to use right version of jdk
  • ls /usr/local/bin/gdb: Need to know gdb location or "which gdb"
  • export DEBUG_PROG=/usr/local/bin/gdb: Or wherever "gdb" is located)
  • <path to java command>/java corefile

Now you will be in the debugger. Execute the following commands:

  • (gdb) where: Shows a summary of the stack
  • (gdb) thr: Switch among threads or show the current thread
  • (gdb) info thr: Inquire about existing threads
  • (gdb) thread apply 1 bt: Apply a command to a list of threads, specifically the backtrace to thread #1)
  • (gdb) quit

If you don't have access to a debugger you can check to see if you have the "pstack" and "pmap" utilities on your operating system. If you do have those utilities (on some operating systems you have to download these utilities separately); you can run them on the system core file to gather information for support. The syntax of the command would be something like this:
$ /usr/proc/bin/pstack core
$ /usr/proc/bin/pmap core

Further Information
You can also go to http://support.bea.com and find some published solutions on core files. In the "Question" field type in "S-16147" to display some additional information about using pstack and pmap on Solaris.

Enabling/Disabling Dr. Watson
The drwtsn32.log files are similar to system core files on Unix. On Windows 2000, these files are found in the following directory: C:\Documents and Settings\All Users\Documents\DrWatson. After entering "drwtsn32 ?" in a DOS command window and hitting enter, the "Dr. Watson for Windows 2000" box appears. The DrWatson log file overview option will display a screen that explains the format of the drwtsn32.log files.

By default, Dr. Watson will be enabled when Windows NT is installed. To disable Dr. Watson, the following Registry value must be changed from 1 to 0 (zero): "\HKEY_LOCAL_MACHINE\SOFTWARE \Microsoft\Windows NT\CurrentVersion\AeDebug". An entry called "Auto" corresponds to how Dr. Watson will start up. To enable Dr. Watson, change the "Auto" value from 0 (zero) to 1. This will then launch whatever debugger, or application, is under the Debugger registry value. For Dr. Watson, the Debugger value should contain:

drwtsn32 -p %ld -e %ld -g

Links to On-Line Debugger Manuals

  1. gdb: www.gnu.org/manual/gdb-5.1.1/gdb.html
  2. dbx:
    a. Sun: http://docs.sun.com/db/doc/805-4948?q=DBX
    b. IBM: http://publib16.boulder.ibm.com/pseries/en_US/cmds/aixcmds2/dbx.htm
  3. adb: HP: http://docs.hp.com/hpux/onlinedocs/B2355-90680/00/00/8-con.html

If none of these hints help direct you towards a solution or an identifier in your application, then you should contact BEA Customer Support for further diagnosis. You can open a case with a valid support contract by logging in at http://support.bea.com/login.jsp.

More Stories By Steve Pozarycki

Steven Pozarycki is a backline developer relations engineer with BEA Systems, Customer Support Group. He specializes in troubleshooting and solving complex customer issues with their mission-critical applications on BEA products. Steve holds a bachelor's degree in computer science.

Reproduced with permission from BEA Systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the applic...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, will introduce two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a...
SYS-CON Events announced today that Calligo has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo is an innovative cloud service provider offering mid-sized companies the highest levels of data privacy. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalized support service from its globally located cloud platform...
SYS-CON Events announced today that Elastifile will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Elastifile Cloud File System (ECFS) is software-defined data infrastructure designed for seamless and efficient management of dynamic workloads across heterogeneous environments. Elastifile provides the architecture needed to optimize your hybrid cloud environment, by facilitating efficient...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Launched in 2016, Cloudistics helps anyone bring the power of the cloud to the data center in an easy-to-use, on- premises cloud platform that automatically provides high performance resources for all types of applications: Docke...
SYS-CON Events announced today that Golden Gate University will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Since 1901, non-profit Golden Gate University (GGU) has been helping adults achieve their professional goals by providing high quality, practice-based undergraduate and graduate educational programs in law, taxation, business and related professions. Many of its courses are taug...
SYS-CON Events announced today that Golden Gate University will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Since 1901, non-profit Golden Gate University (GGU) has been helping adults achieve their professional goals by providing high quality, practice-based undergraduate and graduate educational programs in law, taxation, business and related professions. Many of its courses are taug...
SYS-CON Events announced today that Secure Channels, a cybersecurity firm, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Secure Channels, Inc. offers several products and solutions to its many clients, helping them protect critical data from being compromised and access to computer networks from the unauthorized. The company develops comprehensive data encryption security strategie...
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interopera...
SYS-CON Events announced today that SkyScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SkyScale is a world-class provider of cloud-based, ultra-fast multi-GPU hardware platforms for lease to customers desiring the fastest performance available as a service anywhere in the world. SkyScale builds, configures, and manages dedicated systems strategically located in maximum-security...
SYS-CON Events announced today that Datera will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera offers a radically new approach to data management, where innovative software makes data infrastructure invisible, elastic and able to perform at the highest level. It eliminates hardware lock-in and gives IT organizations the choice to source x86 server nodes, with business model option...
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Me...
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
SYS-CON Events announced today that Akvelon will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Akvelon is a business and technology consulting firm that specializes in applying cutting-edge technology to problems in fields as diverse as mobile technology, sports technology, finance, and healthcare.
SYS-CON Events announced today that Secure Channels, a cybersecurity firm, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Secure Channels, Inc. offers several products and solutions to its many clients, helping them protect critical data from being compromised and access to computer networks from the unauthorized. The company develops comprehensive data encryption security strategie...
SYS-CON Events announced today that WineSOFT will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Based in Seoul and Irvine, WineSOFT is an innovative software house focusing on internet infrastructure solutions. The venture started as a bootstrap start-up in 2010 by focusing on making the internet faster and more powerful. WineSOFT’s knowledge is based on the expertise of TCP/IP, VPN, SS...
WebRTC is the future of browser-to-browser communications, and continues to make inroads into the traditional, difficult, plug-in web communications world. The 6th WebRTC Summit continues our tradition of delivering the latest and greatest presentations within the world of WebRTC. Topics include voice calling, video chat, P2P file sharing, and use cases that have already leveraged the power and convenience of WebRTC.