Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Liquid Data: XQuery-Based Enterprise Information Integration

Liquid Data: XQuery-Based Enterprise Information Integration

Modern enterprises are drowning in a sea of information. Despite owning an ever-increasing volume of information, most enterprises cannot exploit it to even a fraction of its full potential.

This is because the information is strewn across many systems with diverse data formats and interfaces - systems that are largely unaware of one another and of the relationships their content has with information contained elsewhere. Therein lies the Enterprise Information Integration (EII) problem: enterprises need the ability to easily access information about a given business entity, such as a customer or an order, from a varied and distributed collection of information sources.

The EII problem isn't new - the database world has been struggling with it for over two decades - but today's business trends bring a new urgency to finding a solution. Luckily, we are also at a point today where an alignment of emerging technologies is finally enabling a solution. This article looks at why this is so and how BEA's Liquid Data for WebLogic exploits this alignment to offer a commercial EII solution today.

An Example Integration Scenario
Picture a large telecommunications vendor with a Wireless Division and a Broadband Division. Each division has its own order management system and its own customer support system. The vendor also has a centralized CRM system that maintains information about customer purchasing histories and product promotions. A closer look at the vendor's IT environment might reveal the following mix of back-end systems:

Wireless Division:
Order Management System: SAP ERP using Oracle as the RDBMS
Customer Support System: Clarify using Oracle as the RDBMS

Broadband Division:
Order Management System: PeopleSoft ERP using Sybase as the RDBMS
Customer Support System: Homegrown system with SQL Server as the RDBMS

Corporate CRM System:
Siebel using DB2 as the RDBMS

As you can see, customer-related information is spread across five different information systems. (In real situations, the overall number of systems in an enterprise's IT environment would likely be closer to the 10-100 range.) These systems use a mix of technologies from different independent software vendors, and the five systems are unaware of one another. Some allow direct database access, while others allow information access only through their business-level APIs.

The vendor wants to improve its customer responsiveness and operating efficiency by giving customers up-to-the-minute access to all of their account information via several channels - a phone tree system, live telephone customer support representatives, the customer service desk at its sales outlets, and a self-service Web portal. The vendor also hopes to boost sales by adding recommendation features to all four channels. To accomplish this, the vendor wants to create a comprehensive 360º "single view of customer" that can be shared across the channels. To meet the up-to-the-minute goal, the view's data must be obtained directly from the vendor's operational systems. This is a clear instance of the EII problem!

Current EII "Solutions"
To solve this problem, application developers face three basic engineering challenges:

  • The Metadata Challenge: The metadata describing the information contained in different back-end systems is captured in various forms, including RDBMS system catalogs, WSDL files, application views (for J2EE-CA adaptors), and XML Schemas.
  • The Data Access API Challenge: Information contained in different back-end systems must be accessed using the appropriate protocols, such as JDBC/SQL, ODBC/SQL, OCI/SQL (for Oracle data), SOAP, JCA, etc.
  • The Data Model Challenge: The basic format of the data returned from a given back-end system depends on the nature of that system. Possible formats include relational rowsets, Java objects, and XML documents.

    Table 1 lists four common approaches to tackling the EII problem today and briefly analyzes their main strengths and weaknesses. None of these approaches offers a full EII solution.

    Liquid Data: An XML-Based EII Solution
    In November of 2002, BEA introduced Liquid Data for WebLogic. Liquid Data provides real-time access and aggregation of data from disparate data sources, improving visibility for front-office applications. "Real-time" refers here to using the most up-to-date information available. Unlike a data warehouse, which houses a preintegrated copy of selected enterprise data that is days or weeks old, Liquid Data provides access to the current state of the enterprise by directly integrating and providing access to the data in the enterprise's various operational systems. Liquid Data's focus is on data visibility, or integrated read access. BEA's WebLogic Integration complements Liquid Data for applications that require update support. It targets front-office applications, which typically require fast access to an integrated view of one or a few business entities at a time. Our 360º view-of-customer scenario is a classic front-office application.

    Liquid Data distinguishes itself from past EII approaches by taking an XML data integration approach. It combines many of the strengths of the EAI and relational data integration approaches without inheriting their weaknesses. Liquid Data provides an XML standards-based view of all enterprise data - enterprise data sources appear either as virtual XML documents or as a collection of functions that take XML parameters and produce XML results. The W3C XML Schema standard is the enterprise data model, and the W3C XML Query (XQuery) draft standard is the declarative language for specifying integrated reusable views and queries over data distributed across the enterprise. XQuery functions model the functional data sources. EAI-style adaptors are used to "XML-ify" packaged applications that don't natively expose XML APIs. Using XML rather than flat tables makes Liquid Data well suited to handling information from complex data sources, and using XML rather than a proprietary EAI object model positions Liquid Data to interact with applications that support standards-based XML and Web service APIs.

    The Liquid Data Architecture
    Figure 1 depicts the architecture of Liquid Data. At the bottom are the various flavors of data sources supported out of the box. Liquid Data natively supports JDBC/SQL sources (Oracle, DB2, Sybase, SQL Server, and PointBase), Web services (both simple RPC-style and complex document-style), XML data files, and application views of packaged or legacy applications (SAP, PeopleSoft, etc.) using BEA WebLogic Integration adaptors. For JDBC/SQL sources, Liquid Data automatically introspects their relational catalogs to produce default XML views of their content. In the case of Web services, it automatically introspects their WSDL descriptions. In addition to these sources, Liquid Data supports custom Java functions to enable developers to write XML plug-ins for data sources that are not natively supported by Liquid Data. Data views (discussed later), once defined, can also be used as Liquid Data data sources.

    The top of Figure 1 shows applications making data query calls involving data views. The Liquid Data world has the following key players and corresponding responsibilities:

  • System administrator: Configures the data sources for the Liquid Data Server.
  • Data architect: Subject matter expert who understands the relevant business entities and data sources. Defines one or more layers of reusable integrated views (data views) of the business entities of interest. Defines stored queries on top of the data views.
  • Application developer: Calls stored queries, which are usually parameterized, from within application code.

    For the system administrator, Liquid Data extends BEA WebLogic Server's Web-based console. For the data architect, Liquid Data provides a tool, DataView Builder, for use in graphically defining data views and stored queries. Liquid Data focuses on parameterized stored queries because they provide a way to simplify application developers' lives while simultaneously protecting the enterprise's operational data stores from ad hoc query traffic. For the application developer, Liquid Data offers an EJB API, a JSP tag library API, and a Web services API for invoking stored queries.

    At the center of Figure 1 is a distributed query execution engine. This engine processes XML queries by calling the underlying data sources, doing the required data transformations and computations, and returning integrated XML results to the application. Caching is also supported. The data architect can indicate that a given query's results may be cached for reuse when the same query or view is reaccessed (with identical parameters) within a specified time period. Not shown in Figure 1, but important, is that Liquid Data supports security policies on stored queries, data views, and base data sources. The Liquid Data server has been implemented as a J2EE application that can be deployed on a BEA WebLogic Server cluster.

    XQuery: Liquid Data's Secret Sauce
    Liquid Data uses XQuery, which is rapidly gaining broad industry support as the declarative language for defining integrated views of an enterprise's many data sources. We give a brief example here to illustrate how an XML query can stitch together data from multiple sources. The XQuery language is detailed at www.w3.org/TR/xquery; Liquid Data's use of XQuery is covered in http://edocs.bea.com/liquiddata/docs10/index.html. The most heavily used XQuery expression in Liquid Data is the FLWR expression, which is analogous to SELECT-FROM-WHERE queries in SQL. A FLWR expression has the following parts:

  • A FOR clause that generates one or more value sequences, binding the values to query variables. The FOR clause in XQuery is similar to the FROM clause in SQL.
  • A LET clause that binds a temporary variable to the result of an expression. The LET clause is similar to the provision of temporary views in some vendors' dialects of SQL.
  • A WHERE clause that contains Boolean predicates that act to restrict the FOR clause's variable bindings. The WHERE clause in XQuery is directly analogous to the WHERE clause in SQL.
  • A RETURN clause that specifies the query's desired XML output. The XQuery RETURN clause is roughly analogous to the SELECT clause in SQL, though the structures that it can specify are a good deal richer than those of SQL.

    Consider our initial EII scenario. Suppose we want to get contact information for a given Wireless Division customer together with information about their orders for Broadband Division products. Listing 1 shows how XQuery can do this. The topmost FOR clause binds a variable to each wireless customer. The LET clause that follows concatenates the customer's first and last names for output formatting. The WHERE clause filters customers by only keeping those whose ID matches a query parameter. Finally, the large RETURN clause specifies the query's output, which consists of customer data from the Wireless Division's database plus the result of a nested query. The nested query pulls order data from the Broadband Division database; its WHERE clause ensures that only the orders for the customer of interest are queried, and its RETURN clause retrieves the desired order information. Listing 2 shows a sample of this query's output.

    DataView Builder UI
    To minimize the need for the data architect to write XQuery views and stored queries by hand, Liquid Data includes a tool called DataView Builder. DataView Builder supports a pictorial, drag-and-drop, mapping-based approach to query design and construction for XQuery. Figure 2 shows an example of the design view, where input data sources are mapped to target data views to graphically construct an XQuery query. DataView Builder also provides a test-view window where the query can be inspected, run, and optionally hand-tuned.

    Programmatic APIs
    Liquid Data provides three APIs that application developers can use to call Liquid Data queries from within application programs. They are:

  • EJB API: This provides a Stateless Session Bean (SLSB) interface to the Liquid Data server. It is a JDBC-like interface that has methods to execute either stored queries (the norm) or ad hoc queries, with or without query parameters, against the integrated enterprise data known to the Liquid Data server.
  • JSP Tag Library: For JSP and portal developers, Liquid Data provides a JSP tag library facade over its SLSB interface. Each SLSB method has a counterpart in the tag library.
  • Web Service Interface: Developers can ask Liquid Data to convert stored queries into Web services. A corresponding WSDL file is produced for each such stored query. Any application that talks to Web services can utilize Liquid Data services through this API.

    Other Features
    The features of LiquidData are too numerous to be covered here. For more detailed information, we encourage you to browse the online documentation at http://edocs.bea.com/liquiddata/docs10/index.html and download and try the product itself, but we will highlight a few of the key additional features here.

    Administration Console
    For system management, Liquid Data extends WebLogic Server's existing administration console. New console-supported activities include:

  • Managing basic Liquid Data server settings
  • Configuring new Liquid Data data sources
  • Configuring the Liquid Data caching subsystem
  • Defining Liquid Data security policies for data sources and data views
  • Monitoring the activity of the Liquid Data server

    Security
    Liquid Data includes support for security policies and supports access control lists (ACLs) for each stored query. Liquid Data's ideas of users and groups are identical to those of BEA WebLogic Server. For ad hoc queries, Liquid Data supports ACLs at the data source level as well.

    Repository
    To manage EII assets, Liquid Data uses a lightweight, server-based repository to store source and target XML schemas, stored queries, WSDL files, code and metadata for custom functions, and so on. The repository is file-based and provides a simple notion of folders.

    Product Combinations
    Liquid Data is an integral part of the overall BEA Platform family. It can be used in combination with any of the existing BEA products. Interesting BEA product combinations include:

  • Liquid Data plus WebLogic Portal: Portal handles Web presentation and personalization while Liquid Data handles enterprise data integration and aggregation.
  • Liquid Data plus WebLogic Integration Adaptors: This enables Liquid Data to integrate data from packaged applications together with relational and Web service data.
  • Liquid Data plus WebLogic Integration: Liquid Data gives workflows an integrated view of enterprise data while the workflows support acting on (e.g., updating) the data.
  • Liquid Data plus WebLogic Workshop: Liquid Data provides an integrated view of enterprise data to Web service-based applications.

    Summary
    This article has presented a brief technical overview of BEA Liquid Data for WebLogic, which takes a fresh, XML-oriented approach to attacking the enterprise information integration problem. It provides a virtual XML document-based view of the enterprise's disparate data sources, and a declarative XQuery-based paradigm is used to create integrated views of the enterprise's key business entities and to define stored queries over those views.

    References

  • Landers, T., and Rosenberg, R., "An Overview of Multibase," Proceedings of the 2nd International Symposium on Distributed Data Bases, Berlin, Germany. North-Holland Publishing Co., September 1982.
  • "BEA Liquid Data for Weblogic: Increasing Visibility in the Distributed Enterprise," BEA Systems White Paper, October 2002.
  • XQuery 1.0: An XML Query Language. www.w3.org/TR/xquery/. W3C Working Draft, August 2002.
  • Curbera, F., et al. "Unraveling the Web Services Web: An Introduction to SOAP, WSDL, and UDDI," IEEE Internet Computing 6(2), March-April 2002.
  • Liquid Data for WebLogic Documentation: http://edocs.bea.com/liquiddata/docs10/index.html
  • Liquid Data for WebLogic Download: http://commerce.bea.com/downloads/liquid_data.jsp
  • Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    @ThingsExpo Stories
    No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
    The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
    Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
    Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
    With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
    In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
    In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
    Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, discussed how they built...
    22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
    22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud ...
    DevOps at Cloud Expo – being held June 5-7, 2018, at the Javits Center in New York, NY – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits,...
    @DevOpsSummit at Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, is co-located with 22nd Cloud Expo | 1st DXWorld Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait...
    Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
    SYS-CON Events announced today that T-Mobile exhibited at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on qua...
    SYS-CON Events announced today that Cedexis will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Cedexis is the leader in data-driven enterprise global traffic management. Whether optimizing traffic through datacenters, clouds, CDNs, or any combination, Cedexis solutions drive quality and cost-effectiveness. For more information, please visit https://www.cedexis.com.
    SYS-CON Events announced today that Google Cloud has been named “Keynote Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Companies come to Google Cloud to transform their businesses. Google Cloud’s comprehensive portfolio – from infrastructure to apps to devices – helps enterprises innovate faster, scale smarter, stay secure, and do more with data than ever before.
    SYS-CON Events announced today that Vivint to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. As a leading smart home technology provider, Vivint offers home security, energy management, home automation, local cloud storage, and high-speed Internet solutions to more than one million customers throughout the United States and Canada. The end result is a smart home solution that sav...
    SYS-CON Events announced today that Opsani will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Opsani is the leading provider of deployment automation systems for running and scaling traditional enterprise applications on container infrastructure.
    SYS-CON Events announced today that Nirmata will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nirmata provides a comprehensive platform, for deploying, operating, and optimizing containerized applications across clouds, powered by Kubernetes. Nirmata empowers enterprise DevOps teams by fully automating the complex operations and management of application containers and its underlying ...
    SYS-CON Events announced today that Opsani to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. Opsani is creating the next generation of automated continuous deployment tools designed specifically for containers. How is continuous deployment different from continuous integration and continuous delivery? CI/CD tools provide build and test. Continuous Deployment is the means by which...