Standards-Based Integration: The Impact of Web Services & J2EE

Mar. 26, 2002 12:00 AM

Standards can redefine a marketplace consider the impact that SQL had on the relational database market. Standards can also create new markets without HTML and HTTP, there would be no World Wide Web.

My thesis here is that Web services and Java 2 Enterprise Edition (J2EE) will have a similarly dramatic impact on application integration advancing the industry from point-to-point integration solutions developed after the fact (which I call integration "in the small") toward standard application containers that are integration-enabled a priori (integration "in the large").

By application integration (or simply integration), I don't just mean Enterprise Application Integration (EAI), which happens on intranets (that is, behind the firewall). I also include business-to-business integration (B2B), wherein the applications from one company directly interconnect with the applications of a business partner across the Internet (or virtual private network).

In fact, EAI and B2B are already converging. Individual business units (BUs) increasingly manage their own IT infrastructure and applications. As a result, Web technologies are now broadly deployed on intranets to give personnel from one BU access to the data and services of another. Therefore, just as the Web unites intranets and the Internet, we can expect eXtensible Markup Language (XML) and Web services (conventions for passing XML documents between applications) to become ubiquitous for both EAI and B2B. (Note: Web services are most easily defined relative to the Web: the Web was about connecting desktop browsers to documents and applications located anywhere on the network. Web services, then, is about extending that platform [HTTP, Secure Sockets Layer (SSL), and so on] for application-to-application communications. Said differently, new applications are increasingly "Web ready" out of the box i.e., designed to support a Web browser front-end. My claim is that future applications will also be "Web integration ready" ready to plug into the emerging "service grid" that will become the basis for integration in the large.

Making the Business Case
Technology alone is never sufficient to transform an industry; there must also be a compelling business case. Today, large companies depend on tens of thousands of applications. Most of these applications operate in silos, interconnecting only with their close peers. At the same time, the rigors of competition are forcing businesses to specialize to focus on only what they can do better than anyone else. But as businesses divest and outsource, they are forced to ever more closely integrate with their supply chains and distribution channels.

Integration "after the fact" is such a pain point that some application vendors now say that the only antidote is to purchase every business application from a single supplier so these applications are pre-integrated. Call this "worst of breed!" First, experience to date would suggest that the promise of preintegration is more marketing hype than reality. Second, and far more important, the notion that any IT-intensive business could rely on a single application provider is absurd. What about verticalization? (No single vendor can have expertise in all industries.) What about the legacy? What about competitive differentiation (via home-grown software)? An IT-intensive business will be nearly as likely to get all of its applications from a single vendor as Boeing or Airbus would be to get the tens of thousands of components necessary to build an airplane (tires, seats, computers, hydraulics, raw aluminum, etc.) from a single manufacturer.

Integration Complexity
Addressing the complexity and cost of integration may well be an IT organization's single biggest need today, and the trends suggest things are only getting worse. Unfortunately, there are no panaceas‹integration is and will remain an inherently hard problem.

In my experience, the lack of a services-based application architecture is the most typical and problematic barrier to application integration. Within many applications, the business logic is intermixed with the presentation logic. (Note: While the term "services-based architecture" may be relatively new, the concepts have been part of modular programming practice for more than two decades. Yet, many new Web applications similarly lack a services-based architecture: consider that many PERL or ColdFusion scripts intermix HTML page construction and SQL processing. J2EE encourages greater modularity by providing JSPs, JavaBeans, and EJBs to abstract page construction, presentation logic, and business logic, respectively.) Integrating such nonmodular applications generally requires an expensive reorganization of the code to separate out the reusable business logic services (think Web services) from the presentation logic.

In lieu of a costly rewrite, integration of nonmodular applications is often limited to data sharing (re-implementing the SQL calls instead of reusing the business service) or screen scraping (reusing the presentation logic as a business service). Data sharing in general refers to reusing the same schemas and tables across multiple applications a good and well used practice. However, data sharing for application integration means exposing the data model of, say, an Enterprise Resource Planning (ERP) application to a Customer Relationship Management (CRM) system. This violates modular programming practice, increasing application complexity (since SQL queries and updates must be rederived across disparate applications) and fragility (since schema changes within the ERP system may break the CRM application). Screen scraping, on the other hand, does reuse the business logic, but is also fragile (since presentation changes may then break remote services). Screen scraping also makes for tedious programming.

Even for beautifully architected applications, however, integration is still hard (albeit less so). For example, customer in one application might refer to a business partner, while customer in another application means an end user. Even when one application's customer is semantically equivalent to another's, there's often an impedance mismatch between their respective representations object versus relational, Java versus COBOL, ASCII versus EPSIDIC, two-line versus three-line addresses, and so on. Impedance mismatches can also be found in application structure synchronous systems like today's Web applications behave differently from asynchronous (message-based) systems, and both differ from batch systems. Another pervasive example of impendance mismatch is the differences in the timeliness of data between an online transaction processing application and a decision-support application.

(Despite the marketing hype, XML and Web services do not resolve impedance mismatch between representations. XML simply defines a common alphabet for constructing words and documents; it's still up to the respective companies and industries to define the vocabularies [the mapping from XML syntax to business semantics] that enable conversations to take place. Some of these vocabularies will be defined "top down" by industry standards bodies and consortia. Other vocabularies will grow "bottom up" [much as natural languages have] as individual companies or small groups of companies specify the Web services interfaces for conducting business with them. XML-based EAI faces the same challenges, albeit within the boundaries of a single company. While tools that map from one vocabulary to another can ease the burden, defining [and resolving] vocabularies is the biggest barrier to achieving integration in the large. Infrastructure like Web services and J2EE adapters is easy by comparison.)

Integration Standards
Application architecture and impedance mismatch are inherent challenges to integration challenges that are independent of the integration technologies. However, the lack of integration standards is also frustrating IT organizations today. Without appropriate standards, integration in the small will remain costly, and integration in the large, impossible. Today, instead of unifying standards, numerous small vendors offer proprietary integration platforms, making for a fragmented market that cannot protect investment. Collectively, the industry must replace these proprietary technologies with standards.

Proprietary Protocols
The litmus test for a proprietary protocol is whether or not the same platform software has to run on both sides of the network. The Web analogy is compelling without standardized HTML and HTTP, the ubiquitous interconnection of heterogeneous Web browsers and heterogeneous Web servers couldn't have happened. Proprietary protocols simply won't work for the scale of integration demanded on the Web. Some of the early B2B solutions were stalled in part by their dependence on proprietary protocols. The proliferation of Web integration standards will also help lower the cost barriers to entry proprietary B2B protocols, like Electronic Data Interchange (EDI) before them, are simply too expensive to foster ubiquity.

(Beware: While XML is indeed a Web standard, XML document- passing conventions can still be highly proprietary. That's why the emerging family of XML/Web services standards Simple Object Access Protocol (SOAP), Web Services Description Language (WSDL), Universal Description, Discovery, and Integration (UDDI), Electronic Business XML Initiative (ebXML), and so on is so essential. Without such standards, users will be unable to mix and match integration solutions as they have Web technologies.)

Proprietary Adapters
Adapters solve the "last mile" problem of integration: they provide the mapping between a newer technology (such as Web services and Java/J2EE) and a "legacy" technology (such as COBOL/CICS). (Note: By using the term "legacy," I intend no insult. Becoming part of the legacy is the goal we set for all new software, and in a real sense, all production applications are part of the legacy.) Even with the emergence of XML and Web services, adapters remain essential, because little of today's legacy is going to be extended to directly support Web services. Moreover, adapters allow tighter integration with the legacy than Web services by, for example, allowing a single transaction or security context to be shared between the new and old applications.

Without a standard model for adapters, it's nearly impossible to get to critical mass. Consider the impact that standardizing on a common database adapter Java Database Connection (JDBC) had on Java's success. The J2EE Connector Architecture (CA) generalizes JDBC to define a universal model for Java adapters. J2EE CA eliminates the n2 cost of each integration vendor having to "one off" their own proprietary adapter for each and every legacy system. More importantly, J2EE CA protects programming investment in integration solutions the same way JDBC protects investment in database applications.

Proprietary APIs and Infrastructure
The integration server hosts the protocols and adapters that make up the "endpoints" of an integration solution. Integration is, of course, harder than simply mapping from one endpoint to another which is why integration servers also provide sophisticated APIs for developing the "glue" necessary to tie together these endpoints. Unfortunately, within today's integration servers most of these APIs are proprietary. Since the cost of developing an integration solution is mostly in the glue programming, the biggest risk to integration projects is long-term dependency on proprietary APIs. These same proprietary APIs prevent ISVs and SIs from reusing their integration investment across different integration servers.

Following is some of the critical integration server glue that's ripe for standardization:

Messaging: Typically the most fundamental part of the integration server. The Java Messaging Service (JMS) standardizes asynchronous communications for Java applications with an API that supports the spectrum of messaging needs: store/forward versus publish/subscribe; hub-and-spoke versus peer-to-peer; unicast versus multicast; and so on. The programming of Java messaging applications is already converging on JMS the same way that database programming converged on JDBC. (In addition to a full JMS implementation, BEA WebLogic also offers the binding from JMS to J2EE adapters that enable asynchronous integration as well as synchronous.)

Messaging and Web service components: Server-side component models dramatically reduce engineering complexity by moving infrastructure decisions out of the business logic and into deployment descriptors. In this way, the business logic that defines what should be done for a particular JMS message or Web service request can be greatly simplified with Enterprise JavaBeans (EJBs). J2EE message-driven EJBs make it easy to program the "sinks" for JMS messages, while WebLogic Server now automates the mapping from a Web services binding to an EJB session bean or JMS queue or topic.

XML processing and transformation: XML and Web services have become so tightly bound to J2EE services (at least within WebLogic), that very efficient XML-standard parsing and transformation technology must be embedded within the platform. Tools that simplify mapping from one XML schema to another, as well as predefined XML transformations (such as mappings to EDI), should similarly be included within the integration server.

Transactions: Transactions ensure truly guaranteed delivery via a two-phase commit protocol. Without transactionally-guaranteed delivery, there's always a window of vulnerability in which failures can cause a message to be dequeued without the corresponding database operation or other application processing having completed. The J2EE standard is the Java Transaction API (JTA), and is a prerequisite for J2EE CA.

(Note: Two-phase commit is unlikely to be the best model for guaranteeing Web services. Under it, a single coordinator resolves the entire transaction, and IT organizations are justifiably reluctant to turn over the coordination of updates to their business-critical data to a business partner. Instead, more loosely coupled alternatives to the two-phase commit, such as the proposed Business Transaction Processing (BTP) standard, will be used to harden Web services. BTP uses sequencing and compensating actions (similar to transaction sagas) to ensure reliable operation for loosely coupled applications.)

Security: Integration servers need to protect network communications (privacy, protection against tampering) as well as reliably identify participating applications (authentication) and ensure that each participant is only granted access to the appropriate services (authorization). With J2EE, platform security is provided by the Java Authentication and Authorization Service (JAAS) and supporting infrastructure like SSL. JAAS is a J2EE CA requirement, while SSL naturally extends to Web services. (Note: Nonrepudiation, the capability to establish the originator of a request after the fact, for example, to guarantee the authenticity of an order, is an area in which the standards are not yet finished.)

Naming: Glue programming depends on naming and directory services. The Java Naming and Directory Interface (JNDI) is the Java API for manipulating local names (those in the address space) as well as for accessing the contents of a Lightweight Directory Access Protocol (LDAP) repository. This infrastructure now extends to Web services: WebLogic Server, for example, automatically generates and publishes a thin Java client consumer as well as WSDL (both on a Web URL and to a UDDI directory) to accommodate easy access to Web services.

Business Process Management (BPM): BPM refers to the higher-level scripting languages and tools for composing workflows computational sequences of tasks, exceptions, and so on. With WebLogic Integration, for example, workflows can be graphically programmed by dragging and dropping standard J2EE building blocks EJBs, JMS messages, Web services, and so on.

Performance, scalability, and reliability: Integration servers are as business-critical as the applications they interconnect. Delivering the deeply intertwined clustering technologies that transparently ensure quality of service automated replication, load balancing, failover, caching, session protection, prioritization, and routing (including data- or content-dependent routing) is both very hard and very expensive. So expensive that the cost of developing business-critical platform software (e.g., operating systems, databases, and application servers) is broadly amortized as the industry consolidates on a small number of long-term winners.

Monitoring and management: Management consoles need to provide a comprehensive operational view into the interworkings of the integration server and its connections to various applications. Moreover, the integration server should offer extensible instrumentation so that collating a line of business view (orders, dollars, etc.) into the integration hub is as straightforward as possible.

Relevance of J2EE
No doubt the above list is incomplete (e.g., I omitted JDBC, which is needed for data sharing). But it nevertheless makes a compelling case that the future of Java-based integration solutions will be built on top of the J2EE platform! To date, J2EE-standard application servers have primarily been used to host new applications (which may access legacy systems), while the task of integrating existing applications has been left to more proprietary integration servers. But consider that J2EE adapters depend directly upon JTA, JAAS, JNDI, and clustering; and indirectly on JMS (for asynchronous invocation), the EJB container (for message-driven EJBs), XML services, and so on. Similarly, Web services depend directly on HTTP servlets, XML processing, and clustering, and indirectly on EJB, JMS, and so on. Add it up; the conclusion is that virtually all of J2EE is essential to integration, provided you believe in the need for standardization of adapters and Web services!

This should not prove too surprising. After all, why should integration programming differ so fundamentally from application programming? Why should expensive infrastructure software be built twice? Or purchased twice? Of course, the J2EE community has also been hard at work extending J2EE to make it a better fit for integration.

As a result, the industry is rapidly coalescing around the Java/J2EE platform and the .NET alternative from Microsoft. Both platforms have a compelling and largely-shared vision of Web services, a vision which is already being proven with direct interoperability testing. (Note: The BEA WebLogic server native SOAP implementation is interoperability-tested against Microsoft .NET, Micrsoft's SOAP Toolkit, GLUE, and IBM WebSphere [which is based on the Apache SOAP implementation]). Some of the additional value proposition for Java is, however, that

Java/J2EE allows development and deployment across virtually all mainstream computing platforms (coverage is, of course, essential to integration).

Java/J2EE APIs are standards that reflect the contributions of hundreds of companies within the Java community.

Web services bindings can be generated transparently for existing Java/J2EE applications (programmers use what they already know), which is key for integration in the large.

The J2EE CA is commercially viable today.

Indeed, software vendors such as PeopleSoft, Siebel, SAP, and others are already working with BEA to deliver standard J2EE adapters for their enterprise applications. In addition, leading systems integrators like Accenture, AMS, CSC, EDS, KPMG, and so on are collaborating with BEA to develop standards-based integration practices around this new framework. Of course, all of the major Java systems vendors BEA, Sun, IBM, Bull, HP, Compaq, NEC, Nokia, Oracle, Unisys, and so on are also working toward leveraging J2EE as a platform for integration. All this gives J2EE-based integration critical mass.

Conclusion
Of course, this new standard integration platform will take time to mature. Web services standards, in particular, continue to evolve: nonrepudiation, guaranteed delivery, and compensating actions are three key areas of standards activity. J2EE-compliant adapters are only now being built. Moreover, vendors like BEA have had to extend the J2EE CA with essential capabilities like bidirectional communications, support for asynchronous processing (via the Java Message Service), and metadata repositories. (Note: BEA is working with the Java Community Process to ensure that these extensions can be appropriately standardized within a future release of J2EE.)

How, then, should organizations best prepare themselves for standards-based integration? By treating integration challenges both tactically and strategically: tactically, by getting the job done with a mix of best-fit standard and proprietary technologies; strategically, by betting on the emerging standards. In particular, I recommend that you avoid long-term or large commitments to proprietary integration frameworks, at least until the transforming impact of standards weeds out the eventual losers.

For those that still have doubts about the applicability of J2EE, we recently witnessed a very similar transformation in application server platforms. Four years ago there were literally dozens of application servers promoting all sorts of proprietary programming models. We said then that J2EE would drive rapid consolidation in this market, and that those who ignored the impact of J2EE would waste their time and money. The majority of those proprietary platforms are now gone. Those that remain have generally redesigned their product around J2EE (or the emerging Windows-centric .NET alternative from Microsoft).

The smart money is on history repeating itself for Web-based application integration. The compelling need for standardization in Web services and Java APIs will drive this market. As with all technology transformations, right now there are compelling opportunities to gain competitive advantage, especially for independent software vendors and systems integrators focused on integration solutions.

Acknowledgements
I am grateful to numerous colleagues at BEA Systems for their thoughtful insights. I am particularly indebted to Rocky Stewart, Tom Mitchell, Pal Takacsi-Nagy, Ajay Patel, Benjamin Renaud, George Kassabgi, and Christina Grenier for their comments on this paper.

About Scott Dietzen
Scott is widely credited with helping put together the J2EE standard, launching the Web application server category, launching the Java Community Process, and driving the Web services collaboration with Microsoft and IBM. Prior to Zimbra, Scott was CTO of BEA Systems where he was the principal architect of the technology strategy for the WebLogic product family.