Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Smoothing the Hand-Off to Production

Strategies for Bridging the Expertise Gap

It is late Monday afternoon and your application is finally going into production. After a year of development and months of QA, it will be live first thing Tuesday morning. The next several days will be critical as customers interact with the new for the first time.

It would be nice to think developers can kick back, or even get started on new development, but that's not going to happen. Unexpected performance degradation and errors will likely crop up, and application support staff won't be able to address them alone.

Right now, J2EE applications management in production is a mixed bag. Some companies have an experienced WebLogic administrator or specialized application support staff. But many companies don't, with production management falling to some combination of general IT application support staff and developers (with developers shouldering more of the burden because they generally have more J2EE experience). It's not an effective or efficient production management strategy. Companies need to grow the expertise of application support staff so they can handle most simple production issues without requiring assistance from developers.

Performance Assurance Before Production
As a developer, you do all you can to ensure that your application will run well in production. During development, as you design and implement the application's functionality, you also consider performance goals. Time and resources are a challenge in every project. And while functional testing usually takes precedence over performance testing in development and QA, the benefits of discovering performance issues at this stage should not be overlooked. Allocating time and resources to test performance pays for itself as you avoid downtime and maintain customer loyalty. Every development and test plan should identify performance criteria that need to be met prior to production.

But profiling your application's performance in the lab is not the same as subjecting it to real user load. In the lab, you generally don't have the resources for a truly comprehensive stress test, so you test the application on a small cluster of machines handling a simulated user load. This is helpful to identify transactional and system-wide bottlenecks that are in need of attention, and to design how the application will behave when it is overloaded or unable to access some part of the system. For example, if a data source is not available (resulting in some kind of "Error 500") can you inform the user that a piece of data is not available and allow them to proceed in another direction instead of simply halting abruptly?

However, performance testing during development and QA, while important, will not address every possible performance problem. In production, your application will be subjected to loads and usage patterns that you could not anticipate. Your application support team needs to be prepared for the unexpected.

The Expertise Gap
Once the application is ready for production, responsibility for the application should ideally be handed over to a WebLogic administrator or the IT application support staff. Unfortunately, many companies don't have experienced BEA WebLogic Server administrators, so developers are pressed into service to fill the gap.

The reasons for this expertise gap are logical enough when you think about it. In the early days of a new business application platform, it's the developers who have had the most experience working with it. Only when many applications are in production does the application support personnel have enough experience to build their own expertise.

The application support team has two goals: to ensure that application availability is maintained based on established service levels, and to plan for the application's continued availability in the future as required by the business. We'll focus on the first goal here, but be aware that capacity planning is a future activity that requires the expertise application support teams need to develop.

Support teams need to ramp up quickly and have a process in place to deal with the crisis of the day. The expertise they need to develop is quite different from that of application developers - it's more about tuning the application server environment, understanding the resource dependencies of J2EE, and configuring the application servers.

Strategies for Handing Off to Production
Developers can either passively wait to be drawn into firefighting minor performance issues, or play a proactive role in helping the application support team learn to maintain the availability of the application. The amount of support required varies with the size and structure of the team, and the depth to which they are expected to diagnose problems. As WebLogic administration staff gain experience and expertise, they will in time become the experts in running your application as optimally as possible in production.

As part of the transition into production, the development team should produce a support document describing the high-level functionality and dependencies of the application - an overview of the application from a systems administration perspective, if you will. This starting point should give even inexperienced support staff a place to start learning about the application. It should document each tier of the application and external dependencies, and include a list of key error log messages.

While your code resides solely on the BEA WebLogic Server, many other servers provide data and functionality to your application. The support document should outline all Web servers, components developed outside your team, and all back-end databases used by the application for data and customer authentication. A diagram depicting the source of the data and the type of communication used between your system and the external data source would be useful. A good description of all the moving parts in your J2EE environment will help to get them up to speed quickly.

To ensure a stable environment, the WebLogic administrator should become the gatekeeper for all application and configuration changes made to the WebLogic Servers in production. To prepare the WebLogic administrator to manage the server, developers should update the support document with a list of applications that need to be deployed to each server, expected response times, and a list of hotspots and potential bottlenecks for the server and applications.

Once provided with this roadmap, the application support team needs to be able to see how it all works in practice. This is one area in which tools can be beneficial.

Tools to Inform and Educate
As a developer, you have many tools available to help you create and test a large application - basic tools like editors, debuggers, and source control tools; and tools that enhance productivity and enable you to do your job better, such as profilers, memory debuggers, and automated testing tools. Application support teams need their own specialized tools to configure, monitor, and diagnose problems with the application in production. Ideally, such tools not only make for more efficient management, but also educate and help develop expertise.

Tools exist for anything from 24x7 application monitoring to deep problem diagnostics tools for code and database issues. But in terms of helping application support teams develop their J2EE expertise, a good place to start is in the area of WebLogic Server management.

The BEA WebLogic Server cluster is really the front door to the application. It runs the code, pools access to the back-end databases, and manages and coordinates use of the shared resources of the system. To help manage the WebLogic Servers, application support should choose a tool that presents information in a well-organized, real-time interface; highlights trouble areas; and provides insight into the cause of problems (see Figure 1).


Tools that provide a domain overview help isolate whether performance issues are specific to one server, or spread across the entire cluster. With the status of each WebLogic Server, the clusters, and each deployed application, the administrator can quickly determine the extent of the performance outage. This initial triage allows the WebLogic administrator to focus on the problem quickly and provides an initial idea about the impact of the situation on customers.

Presentation of information is a key factor in enabling inexperienced administrators to determine the health of their J2EE systems. There are hundreds of metrics available inside WebLogic Server, each a piece of the performance puzzle. Individually these metrics aren't that useful, but when you are able to visualize the relationships, the performance picture comes into focus, and diagnosing the problem is possible. Tools like the BEA WebLogic Console and command-line interfaces that allow you to browse through the metrics are good if you know what the problem is and may even help WebLogic experts find problems. However, while most WebLogic administrators will rely on the console to make changes to a server configuration, they typically need a more intuitive tool to help find which configuration parameter actually needs to be adjusted.

The best way to bring the WebLogic administrator up to speed on the internals of the WebLogic Application Server is to provide them with a real-time performance view of it. Learning an application and finding performance problems is much easier if you can see how the application and WebLogic Server interact. To see the resources usage fluctuate with the process flow between WebLogic components as customer traffic varies is a powerful educational tool. Once visual alarms are added to the picture, the WebLogic administrator is able to detect and diagnose many problems instead of calling development for assistance. Visual alarms should highlight bottlenecks and indicate where the admin should dive deeper into the application or WebLogic server. A picture is worth a thousand words and a real-time performance picture will save hours of investigation.


Another useful capability to look for in an application server management tool is built-in expert advice. This information enables the WebLogic administrator to make informed decisions about the problem and options for resolution. Development teams may still need to be consulted before the administrator takes corrective actions but the availability of context-sensitive advice will improve the decision-making process within the application support team. The right diagnostic tool is critical whether you are a WebLogic veteran or just starting to manage J2EE applications, and integrated expert advice will improve the confidence of the veterans and allow some tasks to be off-loaded to junior staff.

Running J2EE applications in production takes expertise that isn't widely available yet. Developers are the most qualified to diagnose performance and scalability issues at the moment, but instead of fighting fires in production, they should be concentrating on helping application support staff develop the expertise they need to manage their applications.

Simple application documentation is one strategy that will help support teams maintain the availability of the application in production. And to plan future capacity, empirical data collected from the production application is needed. For day-to-day management (and education), a real-time performance viewing tool is ideal, one that can show the live application in operation and help gather data for capacity planning.

More Stories By Hugh Docherty

Hugh Docherty is a product manager for Quest Software’s J2EE performance management solutions. Hugh has extensive experience managing development of web-based financial applications, on top of ten years experience building and supporting web applications.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

IoT & Smart Cities Stories
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Cell networks have the advantage of long-range communications, reaching an estimated 90% of the world. But cell networks such as 2G, 3G and LTE consume lots of power and were designed for connecting people. They are not optimized for low- or battery-powered devices or for IoT applications with infrequently transmitted data. Cell IoT modules that support narrow-band IoT and 4G cell networks will enable cell connectivity, device management, and app enablement for low-power wide-area network IoT. B...
The hierarchical architecture that distributes "compute" within the network specially at the edge can enable new services by harnessing emerging technologies. But Edge-Compute comes at increased cost that needs to be managed and potentially augmented by creative architecture solutions as there will always a catching-up with the capacity demands. Processing power in smartphones has enhanced YoY and there is increasingly spare compute capacity that can be potentially pooled. Uber has successfully ...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...