Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Weblogic

Weblogic: Article

Liquid Data: XQuery-Based Enterprise Information Integration

Liquid Data: XQuery-Based Enterprise Information Integration

Modern enterprises are drowning in a sea of information. Despite owning an ever-increasing volume of information, most enterprises cannot exploit it to even a fraction of its full potential.

This is because the information is strewn across many systems with diverse data formats and interfaces - systems that are largely unaware of one another and of the relationships their content has with information contained elsewhere. Therein lies the Enterprise Information Integration (EII) problem: enterprises need the ability to easily access information about a given business entity, such as a customer or an order, from a varied and distributed collection of information sources.

The EII problem isn't new - the database world has been struggling with it for over two decades - but today's business trends bring a new urgency to finding a solution. Luckily, we are also at a point today where an alignment of emerging technologies is finally enabling a solution. This article looks at why this is so and how BEA's Liquid Data for WebLogic exploits this alignment to offer a commercial EII solution today.

An Example Integration Scenario
Picture a large telecommunications vendor with a Wireless Division and a Broadband Division. Each division has its own order management system and its own customer support system. The vendor also has a centralized CRM system that maintains information about customer purchasing histories and product promotions. A closer look at the vendor's IT environment might reveal the following mix of back-end systems:

Wireless Division:
Order Management System: SAP ERP using Oracle as the RDBMS
Customer Support System: Clarify using Oracle as the RDBMS

Broadband Division:
Order Management System: PeopleSoft ERP using Sybase as the RDBMS
Customer Support System: Homegrown system with SQL Server as the RDBMS

Corporate CRM System:
Siebel using DB2 as the RDBMS

As you can see, customer-related information is spread across five different information systems. (In real situations, the overall number of systems in an enterprise's IT environment would likely be closer to the 10-100 range.) These systems use a mix of technologies from different independent software vendors, and the five systems are unaware of one another. Some allow direct database access, while others allow information access only through their business-level APIs.

The vendor wants to improve its customer responsiveness and operating efficiency by giving customers up-to-the-minute access to all of their account information via several channels - a phone tree system, live telephone customer support representatives, the customer service desk at its sales outlets, and a self-service Web portal. The vendor also hopes to boost sales by adding recommendation features to all four channels. To accomplish this, the vendor wants to create a comprehensive 360º "single view of customer" that can be shared across the channels. To meet the up-to-the-minute goal, the view's data must be obtained directly from the vendor's operational systems. This is a clear instance of the EII problem!

Current EII "Solutions"
To solve this problem, application developers face three basic engineering challenges:

  • The Metadata Challenge: The metadata describing the information contained in different back-end systems is captured in various forms, including RDBMS system catalogs, WSDL files, application views (for J2EE-CA adaptors), and XML Schemas.
  • The Data Access API Challenge: Information contained in different back-end systems must be accessed using the appropriate protocols, such as JDBC/SQL, ODBC/SQL, OCI/SQL (for Oracle data), SOAP, JCA, etc.
  • The Data Model Challenge: The basic format of the data returned from a given back-end system depends on the nature of that system. Possible formats include relational rowsets, Java objects, and XML documents.

    Table 1 lists four common approaches to tackling the EII problem today and briefly analyzes their main strengths and weaknesses. None of these approaches offers a full EII solution.

    Liquid Data: An XML-Based EII Solution
    In November of 2002, BEA introduced Liquid Data for WebLogic. Liquid Data provides real-time access and aggregation of data from disparate data sources, improving visibility for front-office applications. "Real-time" refers here to using the most up-to-date information available. Unlike a data warehouse, which houses a preintegrated copy of selected enterprise data that is days or weeks old, Liquid Data provides access to the current state of the enterprise by directly integrating and providing access to the data in the enterprise's various operational systems. Liquid Data's focus is on data visibility, or integrated read access. BEA's WebLogic Integration complements Liquid Data for applications that require update support. It targets front-office applications, which typically require fast access to an integrated view of one or a few business entities at a time. Our 360º view-of-customer scenario is a classic front-office application.

    Liquid Data distinguishes itself from past EII approaches by taking an XML data integration approach. It combines many of the strengths of the EAI and relational data integration approaches without inheriting their weaknesses. Liquid Data provides an XML standards-based view of all enterprise data - enterprise data sources appear either as virtual XML documents or as a collection of functions that take XML parameters and produce XML results. The W3C XML Schema standard is the enterprise data model, and the W3C XML Query (XQuery) draft standard is the declarative language for specifying integrated reusable views and queries over data distributed across the enterprise. XQuery functions model the functional data sources. EAI-style adaptors are used to "XML-ify" packaged applications that don't natively expose XML APIs. Using XML rather than flat tables makes Liquid Data well suited to handling information from complex data sources, and using XML rather than a proprietary EAI object model positions Liquid Data to interact with applications that support standards-based XML and Web service APIs.

    The Liquid Data Architecture
    Figure 1 depicts the architecture of Liquid Data. At the bottom are the various flavors of data sources supported out of the box. Liquid Data natively supports JDBC/SQL sources (Oracle, DB2, Sybase, SQL Server, and PointBase), Web services (both simple RPC-style and complex document-style), XML data files, and application views of packaged or legacy applications (SAP, PeopleSoft, etc.) using BEA WebLogic Integration adaptors. For JDBC/SQL sources, Liquid Data automatically introspects their relational catalogs to produce default XML views of their content. In the case of Web services, it automatically introspects their WSDL descriptions. In addition to these sources, Liquid Data supports custom Java functions to enable developers to write XML plug-ins for data sources that are not natively supported by Liquid Data. Data views (discussed later), once defined, can also be used as Liquid Data data sources.

    The top of Figure 1 shows applications making data query calls involving data views. The Liquid Data world has the following key players and corresponding responsibilities:

  • System administrator: Configures the data sources for the Liquid Data Server.
  • Data architect: Subject matter expert who understands the relevant business entities and data sources. Defines one or more layers of reusable integrated views (data views) of the business entities of interest. Defines stored queries on top of the data views.
  • Application developer: Calls stored queries, which are usually parameterized, from within application code.

    For the system administrator, Liquid Data extends BEA WebLogic Server's Web-based console. For the data architect, Liquid Data provides a tool, DataView Builder, for use in graphically defining data views and stored queries. Liquid Data focuses on parameterized stored queries because they provide a way to simplify application developers' lives while simultaneously protecting the enterprise's operational data stores from ad hoc query traffic. For the application developer, Liquid Data offers an EJB API, a JSP tag library API, and a Web services API for invoking stored queries.

    At the center of Figure 1 is a distributed query execution engine. This engine processes XML queries by calling the underlying data sources, doing the required data transformations and computations, and returning integrated XML results to the application. Caching is also supported. The data architect can indicate that a given query's results may be cached for reuse when the same query or view is reaccessed (with identical parameters) within a specified time period. Not shown in Figure 1, but important, is that Liquid Data supports security policies on stored queries, data views, and base data sources. The Liquid Data server has been implemented as a J2EE application that can be deployed on a BEA WebLogic Server cluster.

    XQuery: Liquid Data's Secret Sauce
    Liquid Data uses XQuery, which is rapidly gaining broad industry support as the declarative language for defining integrated views of an enterprise's many data sources. We give a brief example here to illustrate how an XML query can stitch together data from multiple sources. The XQuery language is detailed at www.w3.org/TR/xquery; Liquid Data's use of XQuery is covered in http://edocs.bea.com/liquiddata/docs10/index.html. The most heavily used XQuery expression in Liquid Data is the FLWR expression, which is analogous to SELECT-FROM-WHERE queries in SQL. A FLWR expression has the following parts:

  • A FOR clause that generates one or more value sequences, binding the values to query variables. The FOR clause in XQuery is similar to the FROM clause in SQL.
  • A LET clause that binds a temporary variable to the result of an expression. The LET clause is similar to the provision of temporary views in some vendors' dialects of SQL.
  • A WHERE clause that contains Boolean predicates that act to restrict the FOR clause's variable bindings. The WHERE clause in XQuery is directly analogous to the WHERE clause in SQL.
  • A RETURN clause that specifies the query's desired XML output. The XQuery RETURN clause is roughly analogous to the SELECT clause in SQL, though the structures that it can specify are a good deal richer than those of SQL.

    Consider our initial EII scenario. Suppose we want to get contact information for a given Wireless Division customer together with information about their orders for Broadband Division products. Listing 1 shows how XQuery can do this. The topmost FOR clause binds a variable to each wireless customer. The LET clause that follows concatenates the customer's first and last names for output formatting. The WHERE clause filters customers by only keeping those whose ID matches a query parameter. Finally, the large RETURN clause specifies the query's output, which consists of customer data from the Wireless Division's database plus the result of a nested query. The nested query pulls order data from the Broadband Division database; its WHERE clause ensures that only the orders for the customer of interest are queried, and its RETURN clause retrieves the desired order information. Listing 2 shows a sample of this query's output.

    DataView Builder UI
    To minimize the need for the data architect to write XQuery views and stored queries by hand, Liquid Data includes a tool called DataView Builder. DataView Builder supports a pictorial, drag-and-drop, mapping-based approach to query design and construction for XQuery. Figure 2 shows an example of the design view, where input data sources are mapped to target data views to graphically construct an XQuery query. DataView Builder also provides a test-view window where the query can be inspected, run, and optionally hand-tuned.

    Programmatic APIs
    Liquid Data provides three APIs that application developers can use to call Liquid Data queries from within application programs. They are:

  • EJB API: This provides a Stateless Session Bean (SLSB) interface to the Liquid Data server. It is a JDBC-like interface that has methods to execute either stored queries (the norm) or ad hoc queries, with or without query parameters, against the integrated enterprise data known to the Liquid Data server.
  • JSP Tag Library: For JSP and portal developers, Liquid Data provides a JSP tag library facade over its SLSB interface. Each SLSB method has a counterpart in the tag library.
  • Web Service Interface: Developers can ask Liquid Data to convert stored queries into Web services. A corresponding WSDL file is produced for each such stored query. Any application that talks to Web services can utilize Liquid Data services through this API.

    Other Features
    The features of LiquidData are too numerous to be covered here. For more detailed information, we encourage you to browse the online documentation at http://edocs.bea.com/liquiddata/docs10/index.html and download and try the product itself, but we will highlight a few of the key additional features here.

    Administration Console
    For system management, Liquid Data extends WebLogic Server's existing administration console. New console-supported activities include:

  • Managing basic Liquid Data server settings
  • Configuring new Liquid Data data sources
  • Configuring the Liquid Data caching subsystem
  • Defining Liquid Data security policies for data sources and data views
  • Monitoring the activity of the Liquid Data server

    Security
    Liquid Data includes support for security policies and supports access control lists (ACLs) for each stored query. Liquid Data's ideas of users and groups are identical to those of BEA WebLogic Server. For ad hoc queries, Liquid Data supports ACLs at the data source level as well.

    Repository
    To manage EII assets, Liquid Data uses a lightweight, server-based repository to store source and target XML schemas, stored queries, WSDL files, code and metadata for custom functions, and so on. The repository is file-based and provides a simple notion of folders.

    Product Combinations
    Liquid Data is an integral part of the overall BEA Platform family. It can be used in combination with any of the existing BEA products. Interesting BEA product combinations include:

  • Liquid Data plus WebLogic Portal: Portal handles Web presentation and personalization while Liquid Data handles enterprise data integration and aggregation.
  • Liquid Data plus WebLogic Integration Adaptors: This enables Liquid Data to integrate data from packaged applications together with relational and Web service data.
  • Liquid Data plus WebLogic Integration: Liquid Data gives workflows an integrated view of enterprise data while the workflows support acting on (e.g., updating) the data.
  • Liquid Data plus WebLogic Workshop: Liquid Data provides an integrated view of enterprise data to Web service-based applications.

    Summary
    This article has presented a brief technical overview of BEA Liquid Data for WebLogic, which takes a fresh, XML-oriented approach to attacking the enterprise information integration problem. It provides a virtual XML document-based view of the enterprise's disparate data sources, and a declarative XQuery-based paradigm is used to create integrated views of the enterprise's key business entities and to define stored queries over those views.

    References

  • Landers, T., and Rosenberg, R., "An Overview of Multibase," Proceedings of the 2nd International Symposium on Distributed Data Bases, Berlin, Germany. North-Holland Publishing Co., September 1982.
  • "BEA Liquid Data for Weblogic: Increasing Visibility in the Distributed Enterprise," BEA Systems White Paper, October 2002.
  • XQuery 1.0: An XML Query Language. www.w3.org/TR/xquery/. W3C Working Draft, August 2002.
  • Curbera, F., et al. "Unraveling the Web Services Web: An Introduction to SOAP, WSDL, and UDDI," IEEE Internet Computing 6(2), March-April 2002.
  • Liquid Data for WebLogic Documentation: http://edocs.bea.com/liquiddata/docs10/index.html
  • Liquid Data for WebLogic Download: http://commerce.bea.com/downloads/liquid_data.jsp
  • Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    @ThingsExpo Stories
    In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
    "There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
    "Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
    Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
    Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
    "Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
    "MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
    SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
    "IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
    SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
    "Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
    Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
    It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
    WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
    A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
    SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
    Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
    To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
    An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
    When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...