Welcome!

Weblogic Authors: Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Handling System Core Files

Before you debug

This article will provide some useful tips for debugging your BEA WebLogic Server applications when a system core file occurs. It describes debugging tips, problem troubleshooting, and tools available to assist you in this process.

A system core file is usually indicative of an error in some native code. This could be from the application code of a user (if you are using native code [JNI] in your application), an error in the Java Virtual Machine version you are using, or in BEA WebLogic Server itself. There are a couple of places where native code could have caused the system core file to happen on your operating system. The following ideas and suggestions will help narrow the problem and stabilize the application until the exact cause of the system core can be determined.

The Java Virtual Machine
The first place to look is at the JVM itself. The JVM is a native program and can cause such errors. When in doubt, you can try another certified JVM and/or a later release to determine if a JVM bug is at fault. Many JVM bugs involve the use of the JIT compiler and disabling this feature will often resolve this type of problem. This can be done by supplying the "-Djava.compiler=none" option to the Java command line. Most of the time the JVM will produce a small log file that may contain useful information as to which library the system core may have come from; however, this is not true all of the time. The file is produced in the directory where BEA WebLogic Server was started and is of the form "hs_err_pid<PID #>.log", where "<PID #>" is the process ID of the BEA WebLogic Server process. On AIX this file would be "javacore<PID>.<ID Number>.txt", where "<PID #>" is the process ID of the BEA WebLogic Server process and "<ID Number>" is a number generated by the operating system. You can go to http://java.sun.com and search the Bug Database with the HotSpot "Error ID" number or the method that was happening when the error was reported.

Native Code
The next place to look is at any native libraries that are accessed with JNI (native code) calls in your application, if applicable. This can also cause a system core file to be produced. If your application uses such libraries, they should be carefully examined. It may be difficult to rule out these libraries, as their functionality may not be easily removed from the application. Extensive logging may be needed to determine if a pattern of use can be correlated with the system core dump/Dr. Watson error.

Another place is any Type 2 JDBC driver that makes use of native DBMS libraries, which could also produce this type of error. An option is to switch to a pure Java (Type 4) JDBC driver in order to determine if that is the cause. You can also check with the Type 2 JDBC driver provider to see if there are any known issues or if there is an updated version of the native libraries available.

A final place that could have caused this could be in the BEA WebLogic Server performance pack. This is also native code and when enabled could potentially produce such an error. One option is to disable this feature to determine if that is the cause. This can be done by adding "-Dweblogic.NativeIOEnabled=false" to the command line to start your server. If you are on an older service-pack version you can also check the list of fixed change requests for your particular BEA WebLogic Server version at http://edocs.bea.com.

System Core File
If after looking at the above items you cannot determine the cause of the error, then we can examine the system core file that was produced. If a system core file is produced it will be in the directory where BEA WebLogic Server was started. You will need to run an operating system-specific debugger on the resulting system core file to get a stacktrace that may help in pointing to the offending code.

If you are in development and the system core consistently happens, you can set the following flags to allow a thread dump to be taken of the server right before a core happens. This will get the state of the threads at that moment and may help narrow the problem to application code or point to a bug. The option is "-XX:+ShowMessageBoxOnError" on the Sun JVM (which is not officially documented on the Sun Web site). When the JVM crashes, the program will prompt: "Do you want to debug the problem?" You can then take a thread dump of the JVM. This option will be available on the 8.1 SP2 version of the BEA JRockit JVM when it becomes available. However, in that version the corresponding option will be "-Djrockit.waitonerror".

The best option is to run a debugger on the resulting system core file to get a stacktrace of the native calls. The following information was also given in my last article (WLDJ, Vol. 3, issue 2) but it is being repeated here for consistency. This information may help point out the offending code to you; or, if you are unsure, then contact BEA Customer Support with this information so they can investigate the stacktrace more thoroughly. If you are on a Windows platform, then a "Dr. Watson" file may be produced. Please send this file to BEA Customer Support when opening a case. Otherwise, check the following "Unix" operating system values to make sure that they have already been properly set in order to generate a core file:

  1. Check the "ulimit -c" (configured size of the core file) at a system and user level to make sure that it is set and that the value is not set too low to produce a meaningful core file.
  2. Check the available disk space for the user (for example, is there a disk quota?).
  3. Also check the following parameter, which on Solaris is usually in /etc/system file and can be used to disable core files: set sys:coredumpsize=0.
  4. On Linux, the core dump is turned off by default on all systems. In RedHat Advanced Server 2.1 it should be under "/etc/security". There should be a self-explanatory file called limits.conf; within that file look for the word "core". If it is set to "0", then coredump files are disabled.
  5. On HPUX check the setting called "kernel parm maxdsiz" (max_per_proc_data_size, which increases the User Process Data Segment Size) from the old value of, say, 64M to something higher like 134M.

Please get a stacktrace (or backtrace) from your debugger. Here are the commands needed when using "dbx" or "gdb", which will work on most platforms:

a. dbx:

  • $ java -version: Need to use right version of JDK
  • $ ls /opt/bin/dbx: Need to know dbx location, or "which dbx"
  • $ export: DEBUG_PROG=/opt/bin/dbx (or wherever "dbx" is located)
  • $ <path to java command>/java corefile

Now you will be in the debugger. Execute the following commands:

  • (dbx) where: Shows a summary of the stack
  • (dbx) threads: Shows the state of the existing threads
  • (dbx) quit

b. gdb

  • java -version: Need to use right version of jdk
  • ls /usr/local/bin/gdb: Need to know gdb location or "which gdb"
  • export DEBUG_PROG=/usr/local/bin/gdb: Or wherever "gdb" is located)
  • <path to java command>/java corefile

Now you will be in the debugger. Execute the following commands:

  • (gdb) where: Shows a summary of the stack
  • (gdb) thr: Switch among threads or show the current thread
  • (gdb) info thr: Inquire about existing threads
  • (gdb) thread apply 1 bt: Apply a command to a list of threads, specifically the backtrace to thread #1)
  • (gdb) quit

If you don't have access to a debugger you can check to see if you have the "pstack" and "pmap" utilities on your operating system. If you do have those utilities (on some operating systems you have to download these utilities separately); you can run them on the system core file to gather information for support. The syntax of the command would be something like this:
$ /usr/proc/bin/pstack core
$ /usr/proc/bin/pmap core

Further Information
You can also go to http://support.bea.com and find some published solutions on core files. In the "Question" field type in "S-16147" to display some additional information about using pstack and pmap on Solaris.

Enabling/Disabling Dr. Watson
The drwtsn32.log files are similar to system core files on Unix. On Windows 2000, these files are found in the following directory: C:\Documents and Settings\All Users\Documents\DrWatson. After entering "drwtsn32 ?" in a DOS command window and hitting enter, the "Dr. Watson for Windows 2000" box appears. The DrWatson log file overview option will display a screen that explains the format of the drwtsn32.log files.

By default, Dr. Watson will be enabled when Windows NT is installed. To disable Dr. Watson, the following Registry value must be changed from 1 to 0 (zero): "\HKEY_LOCAL_MACHINE\SOFTWARE \Microsoft\Windows NT\CurrentVersion\AeDebug". An entry called "Auto" corresponds to how Dr. Watson will start up. To enable Dr. Watson, change the "Auto" value from 0 (zero) to 1. This will then launch whatever debugger, or application, is under the Debugger registry value. For Dr. Watson, the Debugger value should contain:

drwtsn32 -p %ld -e %ld -g

Links to On-Line Debugger Manuals

  1. gdb: www.gnu.org/manual/gdb-5.1.1/gdb.html
  2. dbx:
    a. Sun: http://docs.sun.com/db/doc/805-4948?q=DBX
    b. IBM: http://publib16.boulder.ibm.com/pseries/en_US/cmds/aixcmds2/dbx.htm
  3. adb: HP: http://docs.hp.com/hpux/onlinedocs/B2355-90680/00/00/8-con.html

If none of these hints help direct you towards a solution or an identifier in your application, then you should contact BEA Customer Support for further diagnosis. You can open a case with a valid support contract by logging in at http://support.bea.com/login.jsp.

More Stories By Steve Pozarycki

Steven Pozarycki is a backline developer relations engineer with BEA Systems, Customer Support Group. He specializes in troubleshooting and solving complex customer issues with their mission-critical applications on BEA products. Steve holds a bachelor's degree in computer science.

Reproduced with permission from BEA Systems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
"IoT is going to be a huge industry with a lot of value for end users, for industries, for consumers, for manufacturers. How can we use cloud to effectively manage IoT applications," stated Ian Khan, Innovation & Marketing Manager at Solgeniakhela, in this SYS-CON.tv interview at @ThingsExpo, held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
"We're a cybersecurity firm that specializes in engineering security solutions both at the software and hardware level. Security cannot be an after-the-fact afterthought, which is what it's become," stated Richard Blech, Chief Executive Officer at Secure Channels, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Information technology is an industry that has always experienced change, and the dramatic change sweeping across the industry today could not be truthfully described as the first time we've seen such widespread change impacting customer investments. However, the rate of the change, and the potential outcomes from today's digital transformation has the distinct potential to separate the industry into two camps: Organizations that see the change coming, embrace it, and successful leverage it; and...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
Internet of @ThingsExpo, taking place June 6-8, 2017 at the Javits Center in New York City, New York, is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo New York Call for Papers is now open.
What happens when the different parts of a vehicle become smarter than the vehicle itself? As we move toward the era of smart everything, hundreds of entities in a vehicle that communicate with each other, the vehicle and external systems create a need for identity orchestration so that all entities work as a conglomerate. Much like an orchestra without a conductor, without the ability to secure, control, and connect the link between a vehicle’s head unit, devices, and systems and to manage the ...
We are always online. We access our data, our finances, work, and various services on the Internet. But we live in a congested world of information in which the roads were built two decades ago. The quest for better, faster Internet routing has been around for a decade, but nobody solved this problem. We’ve seen band-aid approaches like CDNs that attack a niche's slice of static content part of the Internet, but that’s it. It does not address the dynamic services-based Internet of today. It does...
20th Cloud Expo, taking place June 6-8, 2017, at the Javits Center in New York City, NY, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy.
Everyone knows that truly innovative companies learn as they go along, pushing boundaries in response to market changes and demands. What's more of a mystery is how to balance innovation on a fresh platform built from scratch with the legacy tech stack, product suite and customers that continue to serve as the business' foundation. In his General Session at 19th Cloud Expo, Michael Chambliss, Head of Engineering at ReadyTalk, discussed why and how ReadyTalk diverted from healthy revenue and mor...
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
You have great SaaS business app ideas. You want to turn your idea quickly into a functional and engaging proof of concept. You need to be able to modify it to meet customers' needs, and you need to deliver a complete and secure SaaS application. How could you achieve all the above and yet avoid unforeseen IT requirements that add unnecessary cost and complexity? You also want your app to be responsive in any device at any time. In his session at 19th Cloud Expo, Mark Allen, General Manager of...
The Internet of Things (IoT) promises to simplify and streamline our lives by automating routine tasks that distract us from our goals. This promise is based on the ubiquitous deployment of smart, connected devices that link everything from industrial control systems to automobiles to refrigerators. Unfortunately, comparatively few of the devices currently deployed have been developed with an eye toward security, and as the DDoS attacks of late October 2016 have demonstrated, this oversight can ...
"ReadyTalk is an audio and web video conferencing provider. We've really come to embrace WebRTC as the platform for our future of technology," explained Dan Cunningham, CTO of ReadyTalk, in this SYS-CON.tv interview at WebRTC Summit at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
Major trends and emerging technologies – from virtual reality and IoT, to Big Data and algorithms – are helping organizations innovate in the digital era. However, to create real business value, IT must think beyond the ‘what’ of digital transformation to the ‘how’ to harness emerging trends, innovation and disruption. Architecture is the key that underpins and ties all these efforts together. In the digital age, it’s important to invest in architecture, extend the enterprise footprint to the cl...
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
"Dice has been around for the last 20 years. We have been helping tech professionals find new jobs and career opportunities," explained Manish Dixit, VP of Product and Engineering at Dice, in this SYS-CON.tv interview at 19th Cloud Expo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smar...
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...
Successful digital transformation requires new organizational competencies and capabilities. Research tells us that the biggest impediment to successful transformation is human; consequently, the biggest enabler is a properly skilled and empowered workforce. In the digital age, new individual and collective competencies are required. In his session at 19th Cloud Expo, Bob Newhouse, CEO and founder of Agilitiv, drew together recent research and lessons learned from emerging and established compa...
Businesses and business units of all sizes can benefit from cloud computing, but many don't want the cost, performance and security concerns of public cloud nor the complexity of building their own private clouds. Today, some cloud vendors are using artificial intelligence (AI) to simplify cloud deployment and management. In his session at 20th Cloud Expo, Ajay Gulati, Co-founder and CEO of ZeroStack, will discuss how AI can simplify cloud operations. He will cover the following topics: why clou...