In our earlier article (WLDJ, Vol. 3, issue 5), we discussed the importance of designing for manageability. Using a case study of an on-line shopping application, DizzyWorld, we showed the developer's perspective around application manageability. This included both the development of a JMX MBean and the instrumentation of a Java Page Flow on the BEA WebLogic Platform.

In this article, we look at manageability from the operator's perspective. We will show how management metrics can be defined and mapped to JMX MBeans, and how those metrics can be collected and monitored with HP OpenView. The end goal is to better illustrate how manageability can be designed into your application, and how this data can then be leveraged in building a truly adaptive IT environment. We will begin with background on how the HP OpenView architecture supports the management of the BEA WebLogic platform.

Background
Let's go through a quick introduction of HP OpenView, which consists of a suite of management products. For this example, we will use HP OpenView Operations (OVO) and the Smart Plug-in (SPI) for the BEA WebLogic Server. IT operators can use OVO and various SPIs to manage the entire IT infrastructure. This includes managing clusters of J2EE servers, databases, operating systems and other IT resources. In some cases OVO allows the capability of correlating management data between all of these systems and components. In this scenario, we will leverage OVO in combination with the SPI to provide a management solution that allows an operator to proactively monitor and manage the DizzyWorld application running on the WebLogic platform.

There are three basic components in the OVO architecture: agents, a management server, and display stations (see Figure 1). The agents are responsible for interacting with each managed resource. A management server consists of several managers to proactively send and display messages based on events captured by the agents. The display stations then provide a single view into the managed environment.

OVO monitors, controls, and reports on the health of the IT environment by using different management technologies, including SNMP, ARM, WBEM, CIM, and JMX. OVO and the WebLogic SPI leverage managed resources that are exposed with JMX on the WebLogic Platform. Operators can manage and monitor any WebLogic instance in addition to any JMX-enabled application running on those servers. While performing diagnostics, operators can start and stop WebLogic or turn on tracing and monitoring for a particular WebLogic Server instance. In addition, they can easily discover a newly deployed WebLogic instance running on machines in the network and add it to their management view. This can save valuable time for the network or system administrator.

In addition, the SPI provides access to over 50 different predefined metrics that can be used for management purposes. Developers can use these metrics to get information such as average execution time for a servlet, the number of times per minute that EJB beans were not available from the free pool, or the percentage of the JMS queue that has been filled. These metrics are defined through the JMX MBeans provided by the WebLogic platform.

Operators use these metrics to create reports and graphs or to monitor certain aspects of the WebLogic platform. These monitoring capabilities are based on the concept of OVO policies. The settings within these policies define various conditions and thresholds to monitor within the WebLogic Server. Once in use, the WebLogic SPI policies allow information to be sent back to the OpenView Operations management server to help operators proactively address application problems and avoid serious disruptions to their business.

As Figure 2 shows, the WebLogic SPI performs a number of steps to collect the JMX data for each metric:

1. The OVO Monitor Agent starts based on the scheduling policies defined in OVO.
2. The JMX collector:
   - Accesses a User-defined Metrics (UDM) file that contains references to JMX objects and attributes,
   - Communicates with the MBean Server running within the WebLogic platform and retrieves the data,
   - Performs any calculations and submits the values to the OVO Monitor Agent, where they are compared against defined thresholds.
3. Based on the comparison results, an alarm message may be sent to the OVO console, or an e-mail or other type of notification can be sent to the person responsible for monitoring the application.

Configuring a User-Defined Metric in OpenView
In our previous article, we developed a custom MBean for tracking the state of a shopping cart process and we want to use OVO to monitor it. Assuming this MBean has already been registered with the JMX Server, we will need to define this metric within OVO. The metric can then be used in OVO to generate alarms, create graphs, etc.

The first step is to create a UDM file. The UDM is an XML configuration file that defines what custom JMX MBeans will be collected by OVO. Multiple metrics can be defined in a single UDM file, and each metric can be either a reference to an MBean attribute or a calculated metric that combines other metrics together. Our DizzyWorld application has defined a very simple JMX MBean (see Figure 3).

This MBean defines two attributes, CartsCreatedCount and CartsCompletedCount, which refer to the total number of shopping carts that have been created and completed. We first want to define the mapping of each of these attributes within the UDM file. Listing 1 outlines the specific entry that can be added in the UDM file to retrieve the CartsCreatedCount.

This metric is given the name ShoppingCart_Completed. The FromVersion element within the MBean mapping indicates what versions of WebLogic are supported by this metric. The JMX object and attribute name are specified via the <ObjectName> and <Attribute> elements. Note that the object name supports JMX pattern matching. We specify the pattern string, "*:",Name=MyShoppingCartManager" to locate the MBean for our shopping cart manager.

You can take these individual JMX attribute values and combine them to create more useful, coarse-grained metrics. To do this, you have to construct a calculated metric that references the individual MBean values. In our example, we created two metrics, ShoppingCart_Completed and ShoppingCart_Created. These correspond to the CartsCreatedCount and CartsCompletedCount JMX attribute values. Listing 2 shows how a calculated metric can then be defined in the UDM file, corresponding to the percentage of all shopping carts that have been abandoned by users.

In this example, we have given this metric an ID of "WLSSPI_0710", a naming convention used by the WebLogic Server SPI to collect metrics. Our metric will be referenced as "710" during the configuration of the alarms and thresholds in the next section.

Once we've thought about the specific metrics we want to expose in the UDM file, the next step is to actually create the file. While it can be written using any text or XML editor, we recommend using the OpenView JMX Metric Builder, a tool supplied by HP to aid in UDM construction. This Eclipse-based tool provides facilities to connect and browse the JMX server and build metrics using a forms-based editor. Using the tool's MBean Explorer, you can browse the available JMX MBeans that you wish to use in the UDM file. Figure 4 shows a snapshot of our JMX Server with the MyShoppingCartManager MBean and related attributes highlighted.

Finally, using the Metrics Forms View, you can construct both the MBean metrics and the calculated metrics. Figure 5 provides an example of how our calculated metric for the shopping cart metric can be created using the JMX Metric tool.

Configuring OpenView Operations
With the UDM file created, the next step is to set specific threshold values for sending alarms to OpenView, and then to set the frequency with which we wish to collect this metric.

First, we will configure the generation of alarms through threshold settings. For our example, we want to generate an alarm if the percentage of shopping carts closed is at or below 50%. We must create a monitor policy to set the threshold settings for the metric. As shown in Figure 6, we configured the threshold to be a minimum threshold since we are trying to determine when the rate is less than or equal to 50%. Next, we determined what action we want performed when the threshold value is crossed. We could have configured a number of actions when a threshold is crossed, including sending an alarm to OVO, running a command or script, or generating a report. This flexibility is useful if you are working towards becoming an adaptive enterprise in which management data can be used to adapt the infrastructure to meet business demand.

For this example, we configured a message to be sent to the OVO console. As you can see from the Outgoing Message dialog box in Figure 6, we can set the Severity to a number of levels, including Major, Minor, or Critical. This helps the operator determine the importance of the message. In addition, we added instructions to help the operator diagnose and take appropriate actions (see Figure 7). The instructions include possible causes for the error and solutions for resolving it. This level of detail is important when trying to diagnose the problem.

Figure 8 shows a summary of the threshold and actions information. OVO evaluates the metric and when the threshold limit of 50% is reached, the appropriate message is generated in OVO.

Once the threshold settings have been made, the final step is to determine how often we want OVO to collect this metric. We configured the collector to run every minute. Additionally, we customized the collector program to collect data for our ShoppingCart metric.

Let's now turn back to the DizzyWorld application and see how we can manage and monitor the application.

Managing and Monitoring the Application
Our scenario consists of customers using the DizzyWorld application to do online shopping. Each customer would create a shopping cart during their shopping experience. However, not all of these customers complete their shopping at the Web site. Line-of-business managers want to know when the percentage of "completed carts" out of the "created carts" is 50% or lower. We have already created a threshold setting for 50% and we have the OVO data collector running every minute checking the metric against this threshold.

Within OVO, service maps are available that provide a view of your managed environment. Figure 9 displays a service map for WebLogic. Notice that there is no graphical indication of any event that needs attention at the moment in the "Before" picture. At this point, we will assume that a user has completed one shopping experience, so the carts completed and carts created counts are both 1 in the JMX MBean. We will begin a shopping experience, but instead of completing the purchasing process, we will exit the Web site before completing the purchase. That will leave the counts at 1 cart completed and 2 carts created in the JMX MBean. This will automatically fire an OVO event that indicates too few carts are being completed and the service map in OVO will indicate a problem (as shown in the "After" picture in Figure 9).

Since our application has been instrumented with JMX, certain shopping events update the MBean attribute values each time a user proceeds to checkout or places an order. We have configured OVO to collect on a metric that performs a calculation of the completed shopping rate and checks it against a threshold setting. If the threshold is crossed, an alarm is automatically sent to the OVO console.

We can switch to the message view and see a message in OVO indicating the threshold level has been crossed (see Figure 10). This information is important to the line of business manager and helps him determine how his business is doing. He could correlate this information with WebLogic performance data to help him determine why so many shoppers are leaving the site without completing a purchase. In addition, he could select the "Instructions" tab to get further instructions on how to correct the problem.

Now that we have integrated the JMX metrics with OVO, an operator may want to show those results in a graph. Using OVO's Graph Configuration tool, the operator can configure and view a graph of the business metric. OVO gives an operator the option of displaying many types of graphs from the information that has been collected through the metrics. In this example, we chose a bar graph using our Shopping Cart metric (see Figure 11).

We configured and graphed information on the business metric to give ourselves a graphical view of the number of completed shopping carts, allowing us to display that information for a given period. The graph shows data for a fairly simple metric but operators can correlate the Shopping Carts business metric with JVM Memory Heap, Time of Day, or number of users on the Web site, and take proactive steps to improve response time for the customers using the site.

Conclusion
IT organizations are continually looking for opportunities to reduce costs and to improve operational efficiencies. It is clear that deployment and maintenance issues can be a big factor. In the past, developers have typically looked at management as the responsibility of IT Operations. There are clear benefits to designing and developing for manageability, ensuring that IT has the information they need to quickly analyze and respond to both operational and business needs.

In this article, we took a closer look at application manageability from the perspective of the IT operations staff. Specifically, we looked at J2EE manageability using JMX technology. This article showcased how HP OpenView Operations and the WebLogic SPI allow operators to manage the BEA WebLogic Platform more efficiently. With the SPI, IT operations can quickly view, analyze, and report on operational metrics such as EJB, Servlet, and JMS metrics.

There are situations where you may need to look at more than just the J2EE containers or platform components. To truly build an adaptive enterprise, where IT can better reflect the needs of the business, it's critical that operational and business data is available for monitoring and control. Gathering business metrics often requires the development of custom managed objects and artifacts.

We showed how JMX technology can be used to collect these business metrics. Then, using the user-defined metrics capability within OVO, we showed how these JMX objects could be collected from the BEA WebLogic platform. We further showed how to configure OVO policies to monitor these business metrics for certain threshold conditions.

Clearly, diagnosing applications for performance and SLA conformance requires a more end-to-end view of the environment. Management must take into account the client's perspective, all middleware platforms in deployment, as well as any back-end systems and databases. This analysis often requires a "single pane of glass" into the IT infrastructure, where IT operations and staff can correlate management data and perform root cause analysis.

To address these needs, HP provides a suite of Web application management products that support the BEA WebLogic platform (see Figure 12). From an end-user perspective, the HP OpenView Internet Services (OVIS) reports on the availability and response time of synthetic or real user transactions located as close to the user as possible. HP also offers OpenView Transaction Analyzer (OVTA) for tracing transactions from the client to the application server to backend databases. OVTA also provides a passive application monitoring capability, including application server response time and volume metrics. Combined with OVIS, this provides the user with a complete J2EE application server-level management solution.

Looking Ahead
Looking to the future, we see an increased use of Web services, service-oriented architectures, and grid-based networks. Use of these technologies further increases the demand for robust management software. But, management is not simply about reacting to problems in the systems and applications in development. Organizations need more adaptive management solutions where IT is better aligned with the business. One such solution offered by HP is Life Cycle Management for Web Services (LCM4WS). LCM4WS assists enterprises in better managing life-cycle issues such as deployment, versioning, and security of Web services.

There are a number of technical resources currently available to help get you started in application management and adaptive management. If you are a developer, the best place to start is HP's Dev Resource Central (devresource.hp.com). Here you will find a variety of technical papers, articles, software downloads, and forums focused around JMX and application manageability. If you want to learn more about HP OpenView and the products available for managing web applications, check out openview.hp.com/solutions/appm.

Reference