Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Smoothing the Hand-Off to Production

Strategies for Bridging the Expertise Gap

It is late Monday afternoon and your application is finally going into production. After a year of development and months of QA, it will be live first thing Tuesday morning. The next several days will be critical as customers interact with the new for the first time.

It would be nice to think developers can kick back, or even get started on new development, but that's not going to happen. Unexpected performance degradation and errors will likely crop up, and application support staff won't be able to address them alone.

Right now, J2EE applications management in production is a mixed bag. Some companies have an experienced WebLogic administrator or specialized application support staff. But many companies don't, with production management falling to some combination of general IT application support staff and developers (with developers shouldering more of the burden because they generally have more J2EE experience). It's not an effective or efficient production management strategy. Companies need to grow the expertise of application support staff so they can handle most simple production issues without requiring assistance from developers.

Performance Assurance Before Production
As a developer, you do all you can to ensure that your application will run well in production. During development, as you design and implement the application's functionality, you also consider performance goals. Time and resources are a challenge in every project. And while functional testing usually takes precedence over performance testing in development and QA, the benefits of discovering performance issues at this stage should not be overlooked. Allocating time and resources to test performance pays for itself as you avoid downtime and maintain customer loyalty. Every development and test plan should identify performance criteria that need to be met prior to production.

But profiling your application's performance in the lab is not the same as subjecting it to real user load. In the lab, you generally don't have the resources for a truly comprehensive stress test, so you test the application on a small cluster of machines handling a simulated user load. This is helpful to identify transactional and system-wide bottlenecks that are in need of attention, and to design how the application will behave when it is overloaded or unable to access some part of the system. For example, if a data source is not available (resulting in some kind of "Error 500") can you inform the user that a piece of data is not available and allow them to proceed in another direction instead of simply halting abruptly?

However, performance testing during development and QA, while important, will not address every possible performance problem. In production, your application will be subjected to loads and usage patterns that you could not anticipate. Your application support team needs to be prepared for the unexpected.

The Expertise Gap
Once the application is ready for production, responsibility for the application should ideally be handed over to a WebLogic administrator or the IT application support staff. Unfortunately, many companies don't have experienced BEA WebLogic Server administrators, so developers are pressed into service to fill the gap.

The reasons for this expertise gap are logical enough when you think about it. In the early days of a new business application platform, it's the developers who have had the most experience working with it. Only when many applications are in production does the application support personnel have enough experience to build their own expertise.

The application support team has two goals: to ensure that application availability is maintained based on established service levels, and to plan for the application's continued availability in the future as required by the business. We'll focus on the first goal here, but be aware that capacity planning is a future activity that requires the expertise application support teams need to develop.

Support teams need to ramp up quickly and have a process in place to deal with the crisis of the day. The expertise they need to develop is quite different from that of application developers - it's more about tuning the application server environment, understanding the resource dependencies of J2EE, and configuring the application servers.

Strategies for Handing Off to Production
Developers can either passively wait to be drawn into firefighting minor performance issues, or play a proactive role in helping the application support team learn to maintain the availability of the application. The amount of support required varies with the size and structure of the team, and the depth to which they are expected to diagnose problems. As WebLogic administration staff gain experience and expertise, they will in time become the experts in running your application as optimally as possible in production.

As part of the transition into production, the development team should produce a support document describing the high-level functionality and dependencies of the application - an overview of the application from a systems administration perspective, if you will. This starting point should give even inexperienced support staff a place to start learning about the application. It should document each tier of the application and external dependencies, and include a list of key error log messages.

While your code resides solely on the BEA WebLogic Server, many other servers provide data and functionality to your application. The support document should outline all Web servers, components developed outside your team, and all back-end databases used by the application for data and customer authentication. A diagram depicting the source of the data and the type of communication used between your system and the external data source would be useful. A good description of all the moving parts in your J2EE environment will help to get them up to speed quickly.

To ensure a stable environment, the WebLogic administrator should become the gatekeeper for all application and configuration changes made to the WebLogic Servers in production. To prepare the WebLogic administrator to manage the server, developers should update the support document with a list of applications that need to be deployed to each server, expected response times, and a list of hotspots and potential bottlenecks for the server and applications.

Once provided with this roadmap, the application support team needs to be able to see how it all works in practice. This is one area in which tools can be beneficial.

Tools to Inform and Educate
As a developer, you have many tools available to help you create and test a large application - basic tools like editors, debuggers, and source control tools; and tools that enhance productivity and enable you to do your job better, such as profilers, memory debuggers, and automated testing tools. Application support teams need their own specialized tools to configure, monitor, and diagnose problems with the application in production. Ideally, such tools not only make for more efficient management, but also educate and help develop expertise.

Tools exist for anything from 24x7 application monitoring to deep problem diagnostics tools for code and database issues. But in terms of helping application support teams develop their J2EE expertise, a good place to start is in the area of WebLogic Server management.

The BEA WebLogic Server cluster is really the front door to the application. It runs the code, pools access to the back-end databases, and manages and coordinates use of the shared resources of the system. To help manage the WebLogic Servers, application support should choose a tool that presents information in a well-organized, real-time interface; highlights trouble areas; and provides insight into the cause of problems (see Figure 1).


Tools that provide a domain overview help isolate whether performance issues are specific to one server, or spread across the entire cluster. With the status of each WebLogic Server, the clusters, and each deployed application, the administrator can quickly determine the extent of the performance outage. This initial triage allows the WebLogic administrator to focus on the problem quickly and provides an initial idea about the impact of the situation on customers.

Presentation of information is a key factor in enabling inexperienced administrators to determine the health of their J2EE systems. There are hundreds of metrics available inside WebLogic Server, each a piece of the performance puzzle. Individually these metrics aren't that useful, but when you are able to visualize the relationships, the performance picture comes into focus, and diagnosing the problem is possible. Tools like the BEA WebLogic Console and command-line interfaces that allow you to browse through the metrics are good if you know what the problem is and may even help WebLogic experts find problems. However, while most WebLogic administrators will rely on the console to make changes to a server configuration, they typically need a more intuitive tool to help find which configuration parameter actually needs to be adjusted.

The best way to bring the WebLogic administrator up to speed on the internals of the WebLogic Application Server is to provide them with a real-time performance view of it. Learning an application and finding performance problems is much easier if you can see how the application and WebLogic Server interact. To see the resources usage fluctuate with the process flow between WebLogic components as customer traffic varies is a powerful educational tool. Once visual alarms are added to the picture, the WebLogic administrator is able to detect and diagnose many problems instead of calling development for assistance. Visual alarms should highlight bottlenecks and indicate where the admin should dive deeper into the application or WebLogic server. A picture is worth a thousand words and a real-time performance picture will save hours of investigation.


Another useful capability to look for in an application server management tool is built-in expert advice. This information enables the WebLogic administrator to make informed decisions about the problem and options for resolution. Development teams may still need to be consulted before the administrator takes corrective actions but the availability of context-sensitive advice will improve the decision-making process within the application support team. The right diagnostic tool is critical whether you are a WebLogic veteran or just starting to manage J2EE applications, and integrated expert advice will improve the confidence of the veterans and allow some tasks to be off-loaded to junior staff.

Running J2EE applications in production takes expertise that isn't widely available yet. Developers are the most qualified to diagnose performance and scalability issues at the moment, but instead of fighting fires in production, they should be concentrating on helping application support staff develop the expertise they need to manage their applications.

Simple application documentation is one strategy that will help support teams maintain the availability of the application in production. And to plan future capacity, empirical data collected from the production application is needed. For day-to-day management (and education), a real-time performance viewing tool is ideal, one that can show the live application in operation and help gather data for capacity planning.

More Stories By Hugh Docherty

Hugh Docherty is a product manager for Quest Software’s J2EE performance management solutions. Hugh has extensive experience managing development of web-based financial applications, on top of ten years experience building and supporting web applications.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

IoT & Smart Cities Stories
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
Nicolas Fierro is CEO of MIMIR Blockchain Solutions. He is a programmer, technologist, and operations dev who has worked with Ethereum and blockchain since 2014. His knowledge in blockchain dates to when he performed dev ops services to the Ethereum Foundation as one the privileged few developers to work with the original core team in Switzerland.
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...
If a machine can invent, does this mean the end of the patent system as we know it? The patent system, both in the US and Europe, allows companies to protect their inventions and helps foster innovation. However, Artificial Intelligence (AI) could be set to disrupt the patent system as we know it. This talk will examine how AI may change the patent landscape in the years to come. Furthermore, ways in which companies can best protect their AI related inventions will be examined from both a US and...
Bill Schmarzo, Tech Chair of "Big Data | Analytics" of upcoming CloudEXPO | DXWorldEXPO New York (November 12-13, 2018, New York City) today announced the outline and schedule of the track. "The track has been designed in experience/degree order," said Schmarzo. "So, that folks who attend the entire track can leave the conference with some of the skills necessary to get their work done when they get back to their offices. It actually ties back to some work that I'm doing at the University of San...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...