|By Steve Pozarycki||
|March 10, 2004 12:00 AM EST||
This article will provide some useful tips for debugging your BEA WebLogic Server applications when a system core file occurs. It describes debugging tips, problem troubleshooting, and tools available to assist you in this process.
A system core file is usually indicative of an error in some native code. This could be from the application code of a user (if you are using native code [JNI] in your application), an error in the Java Virtual Machine version you are using, or in BEA WebLogic Server itself. There are a couple of places where native code could have caused the system core file to happen on your operating system. The following ideas and suggestions will help narrow the problem and stabilize the application until the exact cause of the system core can be determined.
The Java Virtual Machine
The first place to look is at the JVM itself. The JVM is a native program and can cause such errors. When in doubt, you can try another certified JVM and/or a later release to determine if a JVM bug is at fault. Many JVM bugs involve the use of the JIT compiler and disabling this feature will often resolve this type of problem. This can be done by supplying the "-Djava.compiler=none" option to the Java command line. Most of the time the JVM will produce a small log file that may contain useful information as to which library the system core may have come from; however, this is not true all of the time. The file is produced in the directory where BEA WebLogic Server was started and is of the form "hs_err_pid<PID #>.log", where "<PID #>" is the process ID of the BEA WebLogic Server process. On AIX this file would be "javacore<PID>.<ID Number>.txt", where "<PID #>" is the process ID of the BEA WebLogic Server process and "<ID Number>" is a number generated by the operating system. You can go to http://java.sun.com and search the Bug Database with the HotSpot "Error ID" number or the method that was happening when the error was reported.
The next place to look is at any native libraries that are accessed with JNI (native code) calls in your application, if applicable. This can also cause a system core file to be produced. If your application uses such libraries, they should be carefully examined. It may be difficult to rule out these libraries, as their functionality may not be easily removed from the application. Extensive logging may be needed to determine if a pattern of use can be correlated with the system core dump/Dr. Watson error.
Another place is any Type 2 JDBC driver that makes use of native DBMS libraries, which could also produce this type of error. An option is to switch to a pure Java (Type 4) JDBC driver in order to determine if that is the cause. You can also check with the Type 2 JDBC driver provider to see if there are any known issues or if there is an updated version of the native libraries available.
A final place that could have caused this could be in the BEA WebLogic Server performance pack. This is also native code and when enabled could potentially produce such an error. One option is to disable this feature to determine if that is the cause. This can be done by adding "-Dweblogic.NativeIOEnabled=false" to the command line to start your server. If you are on an older service-pack version you can also check the list of fixed change requests for your particular BEA WebLogic Server version at http://edocs.bea.com.
System Core File
If after looking at the above items you cannot determine the cause of the error, then we can examine the system core file that was produced. If a system core file is produced it will be in the directory where BEA WebLogic Server was started. You will need to run an operating system-specific debugger on the resulting system core file to get a stacktrace that may help in pointing to the offending code.
If you are in development and the system core consistently happens, you can set the following flags to allow a thread dump to be taken of the server right before a core happens. This will get the state of the threads at that moment and may help narrow the problem to application code or point to a bug. The option is "-XX:+ShowMessageBoxOnError" on the Sun JVM (which is not officially documented on the Sun Web site). When the JVM crashes, the program will prompt: "Do you want to debug the problem?" You can then take a thread dump of the JVM. This option will be available on the 8.1 SP2 version of the BEA JRockit JVM when it becomes available. However, in that version the corresponding option will be "-Djrockit.waitonerror".
The best option is to run a debugger on the resulting system core file to get a stacktrace of the native calls. The following information was also given in my last article (WLDJ, Vol. 3, issue 2) but it is being repeated here for consistency. This information may help point out the offending code to you; or, if you are unsure, then contact BEA Customer Support with this information so they can investigate the stacktrace more thoroughly. If you are on a Windows platform, then a "Dr. Watson" file may be produced. Please send this file to BEA Customer Support when opening a case. Otherwise, check the following "Unix" operating system values to make sure that they have already been properly set in order to generate a core file:
- Check the "ulimit -c" (configured size of the core file) at a system and user level to make sure that it is set and that the value is not set too low to produce a meaningful core file.
- Check the available disk space for the user (for example, is there a disk quota?).
- Also check the following parameter, which on Solaris is usually in /etc/system file and can be used to disable core files: set sys:coredumpsize=0.
- On Linux, the core dump is turned off by default on all systems. In RedHat Advanced Server 2.1 it should be under "/etc/security". There should be a self-explanatory file called limits.conf; within that file look for the word "core". If it is set to "0", then coredump files are disabled.
- On HPUX check the setting called "kernel parm maxdsiz" (max_per_proc_data_size, which increases the User Process Data Segment Size) from the old value of, say, 64M to something higher like 134M.
Please get a stacktrace (or backtrace) from your debugger. Here are the commands needed when using "dbx" or "gdb", which will work on most platforms:
- $ java -version: Need to use right version of JDK
- $ ls /opt/bin/dbx: Need to know dbx location, or "which dbx"
- $ export: DEBUG_PROG=/opt/bin/dbx (or wherever "dbx" is located)
- $ <path to java command>/java corefile
Now you will be in the debugger. Execute the following commands:
- (dbx) where: Shows a summary of the stack
- (dbx) threads: Shows the state of the existing threads
- (dbx) quit
- java -version: Need to use right version of jdk
- ls /usr/local/bin/gdb: Need to know gdb location or "which gdb"
- export DEBUG_PROG=/usr/local/bin/gdb: Or wherever "gdb" is located)
- <path to java command>/java corefile
Now you will be in the debugger. Execute the following commands:
- (gdb) where: Shows a summary of the stack
- (gdb) thr: Switch among threads or show the current thread
- (gdb) info thr: Inquire about existing threads
- (gdb) thread apply 1 bt: Apply a command to a list of threads, specifically the backtrace to thread #1)
- (gdb) quit
If you don't have access to a debugger you can check to see if you have the "pstack" and "pmap" utilities on your operating system. If you do have those utilities (on some operating systems you have to download these utilities separately); you can run them on the system core file to gather information for support. The syntax of the command would be something like this:
$ /usr/proc/bin/pstack core
$ /usr/proc/bin/pmap core
You can also go to http://support.bea.com and find some published solutions on core files. In the "Question" field type in "S-16147" to display some additional information about using pstack and pmap on Solaris.
Enabling/Disabling Dr. Watson
The drwtsn32.log files are similar to system core files on Unix. On Windows 2000, these files are found in the following directory: C:\Documents and Settings\All Users\Documents\DrWatson. After entering "drwtsn32 ?" in a DOS command window and hitting enter, the "Dr. Watson for Windows 2000" box appears. The DrWatson log file overview option will display a screen that explains the format of the drwtsn32.log files.
By default, Dr. Watson will be enabled when Windows NT is installed. To disable Dr. Watson, the following Registry value must be changed from 1 to 0 (zero): "\HKEY_LOCAL_MACHINE\SOFTWARE \Microsoft\Windows NT\CurrentVersion\AeDebug". An entry called "Auto" corresponds to how Dr. Watson will start up. To enable Dr. Watson, change the "Auto" value from 0 (zero) to 1. This will then launch whatever debugger, or application, is under the Debugger registry value. For Dr. Watson, the Debugger value should contain:
drwtsn32 -p %ld -e %ld -g
Links to On-Line Debugger Manuals
- gdb: www.gnu.org/manual/gdb-5.1.1/gdb.html
a. Sun: http://docs.sun.com/db/doc/805-4948?q=DBX
b. IBM: http://publib16.boulder.ibm.com/pseries/en_US/cmds/aixcmds2/dbx.htm
- adb: HP: http://docs.hp.com/hpux/onlinedocs/B2355-90680/00/00/8-con.html
If none of these hints help direct you towards a solution or an identifier in your application, then you should contact BEA Customer Support for further diagnosis. You can open a case with a valid support contract by logging in at http://support.bea.com/login.jsp.
The Internet of Everything is re-shaping technology trends–moving away from “request/response” architecture to an “always-on” Streaming Web where data is in constant motion and secure, reliable communication is an absolute necessity. As more and more THINGS go online, the challenges that developers will need to address will only increase exponentially. In his session at @ThingsExpo, Todd Greene, Founder & CEO of PubNub, exploreed the current state of IoT connectivity and review key trends and technology requirements that will drive the Internet of Things from hype to reality.
Nov. 25, 2015 05:45 AM EST Reads: 375
Two weeks ago (November 3-5), I attended the Cloud Expo Silicon Valley as a speaker, where I presented on the security and privacy due diligence requirements for cloud solutions. Cloud security is a topical issue for every CIO, CISO, and technology buyer. Decision-makers are always looking for insights on how to mitigate the security risks of implementing and using cloud solutions. Based on the presentation topics covered at the conference, as well as the general discussions heard between sessions, I wanted to share some of my observations on emerging trends. As cyber security serves as a fou...
Nov. 25, 2015 05:45 AM EST Reads: 287
Continuous processes around the development and deployment of applications are both impacted by -- and a benefit to -- the Internet of Things trend. To help better understand the relationship between DevOps and a plethora of new end-devices and data please welcome Gary Gruver, consultant, author and a former IT executive who has led many large-scale IT transformation projects, and John Jeremiah, Technology Evangelist at Hewlett Packard Enterprise (HPE), on Twitter at @j_jeremiah. The discussion is moderated by me, Dana Gardner, Principal Analyst at Interarbor Solutions.
Nov. 25, 2015 02:30 AM EST Reads: 675
Too often with compelling new technologies market participants become overly enamored with that attractiveness of the technology and neglect underlying business drivers. This tendency, what some call the “newest shiny object syndrome” is understandable given that virtually all of us are heavily engaged in technology. But it is also mistaken. Without concrete business cases driving its deployment, IoT, like many other technologies before it, will fade into obscurity.
Nov. 25, 2015 02:00 AM EST Reads: 283
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound effect on the world, and what should we expect to see over the next couple of years.
Nov. 25, 2015 12:30 AM EST Reads: 410
With all the incredible momentum behind the Internet of Things (IoT) industry, it is easy to forget that not a single CEO wakes up and wonders if “my IoT is broken.” What they wonder is if they are making the right decisions to do all they can to increase revenue, decrease costs, and improve customer experience – effectively the same challenges they have always had in growing their business. The exciting thing about the IoT industry is now these decisions can be better, faster, and smarter. Now all corporate assets – people, objects, and spaces – can share information about themselves and thei...
Nov. 25, 2015 12:00 AM EST Reads: 159
PubNub has announced the release of BLOCKS, a set of customizable microservices that give developers a simple way to add code and deploy features for realtime apps.PubNub BLOCKS executes business logic directly on the data streaming through PubNub’s network without splitting it off to an intermediary server controlled by the customer. This revolutionary approach streamlines app development, reduces endpoint-to-endpoint latency, and allows apps to better leverage the enormous scalability of PubNub’s Data Stream Network.
Nov. 24, 2015 10:00 PM EST Reads: 259
I recently attended and was a speaker at the 4th International Internet of @ThingsExpo at the Santa Clara Convention Center. I also had the opportunity to attend this event last year and I wrote a blog from that show talking about how the “Enterprise Impact of IoT” was a key theme of last year’s show. I was curious to see if the same theme would still resonate 365 days later and what, if any, changes I would see in the content presented.
Nov. 24, 2015 08:00 PM EST Reads: 342
Apps and devices shouldn't stop working when there's limited or no network connectivity. Learn how to bring data stored in a cloud database to the edge of the network (and back again) whenever an Internet connection is available. In his session at 17th Cloud Expo, Ben Perlmutter, a Sales Engineer with IBM Cloudant, demonstrated techniques for replicating cloud databases with devices in order to build offline-first mobile or Internet of Things (IoT) apps that can provide a better, faster user experience, both offline and online. The focus of this talk was on IBM Cloudant, Apache CouchDB, and ...
Nov. 24, 2015 07:30 PM EST Reads: 345
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds. How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem" in this scenario: microservice A (releases daily) depends on a couple of additions to backend B (re...
Nov. 24, 2015 06:00 PM EST Reads: 369
Discussions of cloud computing have evolved in recent years from a focus on specific types of cloud, to a world of hybrid cloud, and to a world dominated by the APIs that make today's multi-cloud environments and hybrid clouds possible. In this Power Panel at 17th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists addressed the importance of customers being able to use the specific technologies they need, through environments and ecosystems that expose their APIs to make true change and transformation possible.
Nov. 24, 2015 03:30 PM EST Reads: 461
There are over 120 breakout sessions in all, with Keynotes, General Sessions, and Power Panels adding to three days of incredibly rich presentations and content. Join @ThingsExpo conference chair Roger Strukhoff (@IoT2040), June 7-9, 2016 in New York City, for three days of intense 'Internet of Things' discussion and focus, including Big Data's indespensable role in IoT, Smart Grids and Industrial Internet of Things, Wearables and Consumer IoT, as well as (new) IoT's use in Vertical Markets.
Nov. 24, 2015 03:30 PM EST Reads: 508
Container technology is shaping the future of DevOps and it’s also changing the way organizations think about application development. With the rise of mobile applications in the enterprise, businesses are abandoning year-long development cycles and embracing technologies that enable rapid development and continuous deployment of apps. In his session at DevOps Summit, Kurt Collins, Developer Evangelist at Built.io, examined how Docker has evolved into a highly effective tool for application delivery by allowing increasingly popular Mobile Backend-as-a-Service (mBaaS) platforms to quickly crea...
Nov. 24, 2015 03:00 PM EST Reads: 289
The Internet of Things (IoT) is growing rapidly by extending current technologies, products and networks. By 2020, Cisco estimates there will be 50 billion connected devices. Gartner has forecast revenues of over $300 billion, just to IoT suppliers. Now is the time to figure out how you’ll make money – not just create innovative products. With hundreds of new products and companies jumping into the IoT fray every month, there’s no shortage of innovation. Despite this, McKinsey/VisionMobile data shows "less than 10 percent of IoT developers are making enough to support a reasonably sized team....
Nov. 24, 2015 02:00 PM EST Reads: 418
The cloud. Like a comic book superhero, there seems to be no problem it can’t fix or cost it can’t slash. Yet making the transition is not always easy and production environments are still largely on premise. Taking some practical and sensible steps to reduce risk can also help provide a basis for a successful cloud transition. A plethora of surveys from the likes of IDG and Gartner show that more than 70 percent of enterprises have deployed at least one or more cloud application or workload. Yet a closer inspection at the data reveals less than half of these cloud projects involve production...
Nov. 24, 2015 01:45 PM EST Reads: 410
As organizations realize the scope of the Internet of Things, gaining key insights from Big Data, through the use of advanced analytics, becomes crucial. However, IoT also creates the need for petabyte scale storage of data from millions of devices. A new type of Storage is required which seamlessly integrates robust data analytics with massive scale. These storage systems will act as “smart systems” provide in-place analytics that speed discovery and enable businesses to quickly derive meaningful and actionable insights. In his session at @ThingsExpo, Paul Turner, Chief Marketing Officer at...
Nov. 24, 2015 01:45 PM EST Reads: 339
Internet of @ThingsExpo, taking place June 7-9, 2016 at Javits Center, New York City and Nov 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 18th International @CloudExpo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo New York Call for Papers is now open.
Nov. 24, 2015 01:30 PM EST Reads: 488
We all know that data growth is exploding and storage budgets are shrinking. Instead of showing you charts on about how much data there is, in his General Session at 17th Cloud Expo, Scott Cleland, Senior Director of Product Marketing at HGST, showed how to capture all of your data in one place. After you have your data under control, you can then analyze it in one place, saving time and resources.
Nov. 24, 2015 01:30 PM EST
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York and Silicon Valley. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place Nov 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 17th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound cha...
Nov. 24, 2015 01:30 PM EST Reads: 482
We are rapidly moving to a brave new world of interconnected smart homes, cars, offices and factories known as the Internet of Things (IoT). Sensors and monitoring devices will touch every part of our lives. Let's take a closer look at the Internet of Things. The Internet of Things is a worldwide network of objects and devices connected to the Internet. They are electronics, sensors, software and more. These objects connect to the Internet and can be controlled remotely via apps and programs. Because they can be accessed via the Internet, these devices create a tremendous opportunity to inte...
Nov. 24, 2015 01:15 PM EST Reads: 482