Welcome!

Weblogic Authors: Yeshim Deniz, Elizabeth White, Michael Meiner, Michael Bushong, Avi Rosenthal

Related Topics: Java IoT, IBM Cloud, Weblogic, Machine Learning

Java IoT: Article

Systems Integration with Openadaptor

Business system integration with little or no custom programming

Examples
Perhaps the best way to demonstrate the ease and simplicity of assembling adaptors is by example. Two adaptors will be constructed, both of which are going to be expressed as Spring framework configuration files (full configuration listings are included as part of the latest Openadaptor distribution).

Before we proceed, setting up an environment to run the examples shown here should be straightforward and not take too long. For the first example all that is required is the Openadaptor distribution (www.openadaptor.org/downloads.html) and a JRE 1.4+ installed on the system. After unzipping the package, the CLASSPATH should be set appropriately to run the examples. The XML configurations presented here, together with detailed instructions on setting the CLASSPATH and running the adaptors, can be found in openadaptor\example\spring\demo\*.xml in the standard Openadaptor distribution. The second example uses a JMS write connector that requires running a (default) JBoss implementation (to provide the JMS service). It also illustrates reading data from a database, for simplicity Hypersonic (a.k.a. HSQLDB) is assumed, and again should be available. Familiarity with JBoss or Hypersonic is not required. See Resources for Openadaptor, JBoss, Hypersonic and JRE download links.

Example One
The first adaptor exposes a servlet interface to receive data, which is then forwarded to an XML convertor that in turn passes it to a console printer (see Figure 4). The adaptor is deliberately contrived and simplistic in order to make it easy to run and experiment with. In the second example it will be replaced with a more realistic scenario. Data may be injected into the adaptor via a test web page. The XML configuration will hold four bean definitions. One of the beans is a servlet (IReadConnector) that will await HTTP requests to deliver input data into the adaptor. When a request comes in, it will be passed to an XML convertor that will turn the tabular data in the form of a Map into an XML document. Last, a write connector will print the resulting XML document on the screen. Listing 1 provides the three bean definitions.

Normally, bean definitions are reasonably terse as Java classes that provide their implementations take sensible defaults. The server bean, when instantiated, will start an HTTP server and listen for requests on port 9999. It will by default convert the data it receives to an ordered map, which will then be passed on to the toXMLConvetor, which will make an attempt to convert it to an XML document. The last component in the pipeline, the file writer, defaults to console (stdout) output if no file name is provided.

The last required definition is that of the adaptor (see Listing 2). The adaptor's messageProcessor property may be used to arrange the components into what is loosely referred to as a processing pipeline.

Before continuing, it's worth noting the optional JMX monitoring feature. Should it be required, it is possible to monitor the state of the adaptor by exposing the components (including the Adaptor) as MBeans that will be accessible via a JMX console. To do so, it suffices to add the following bean definition shown to the configuration.

<bean class="org.openadaptor.core.jmx.MBeanServer">
    <property name="port" value="8082"/>
</bean>

Figure 4 illustrates the structure of the adaptor; note the data from a sample request and the format it takes when passing between components. The payload type, as might be expected, changes from an ordered map to a String containing an XML document in the toXMLConvertor.

The adaptor's XML configuration is now complete. After starting the adaptor from the command line, opening the test page within a web browser will allow test data to be injected (download test page here). After sending a sample request, the adaptor's console output should be similar to the example1output.html. (The source code for this article can be downloaded from the online version of this article at http://java.sys-con.com.)

Example Two
In this example several more advanced features will be presented, such as an enrichment processor, a fan-out (splitting of the processing pipeline into multiple branches), and exception handling. The job of the adaptor will be to read a comma-delimited file, filter out records of interest, enrich them with relevant data stored in a relational database, and finally publish data to a JMS queue. Records that do not have the expected format will be caught by the exception handler and written to an error log file. Apart from writing to a JMS destination, the output data will also be written to a flat file. Reading of the input file will be throttled - records will be read in with a one second interval in between (see Figure 5).

Please note: if you want to run this example in its full version, you will need to download, install, and run the JBoss application server and Hypersonic database engine (see Resources for download links). Optionally, if you cannot or don't want to install JBoss, remove the jmsPublisher component from the processing pipeline. Likewise, if you take out the dbBasedEnricher component, you won't need the Hypersonic database to be available.

The first bean definition will be of a file reader that sources data from a simple comma-delimited file (input/delim.txt) located in the input folder, relative to the location the adaptor is executed from. This is the file's content (note that the second record is missing the last field):

palinm, Michael Palin, 1
chapmang, Graham Chapman
jonest, Terry Jones, 3
idlee, Eric Idle, 4

More Stories By Kris Lachor

Kris Lachor is a senior software developer at Dresdner Kleinwort.

Comments (2) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
higginse 10/28/08 09:41:45 AM EDT

Ismael,
Thanks for the feedback, it's always very useful to us. You should be able use stored procedures for updates - see the MapCallableStatementWriter (and corresponding map-stored-proc-writer.xml example, and their XML equivalents). In fact, if you can guarantee that the warehouse will always be based on Oracle it might be easiest to implement the 'upsert' funtionality in a stored procedure which the JDBC writer would invoke.

Having said that, with your comments in mind, we are looking at adding direct support for updates (and possibly 'upserts') with a view towards squeezing it into our upcoming 3.4.3 (November '08) release of openadaptor.

Also it's worth noting that the RawSQLWriter can achieve the same effect - it uses actual SQL strings which get directly executed against the db, albeit with a performance penalty (overhead of constructing the SQL manually, and not being able to compile it).

regards
Eddy Higgins, Kris Lachor (& the openadaptor dev team)

P.S. Note that www.openadaptor.org is the primary resource for openadaptor technical support. This affords the best visibility to the dev team, and other oa users.

imatos 10/27/08 11:55:00 AM EDT

Great article. Great tool. It seems a good fit to a new project I'm involved with.

The project will require extracting data from a ERP system to load into a data warehouse. The ERP system uses a DB2 UDB database, while the data warehouse will be based on an Oracle database. On a daily basis, it is necessary to extract all the new or updated records from selected tables on our ERP database to load on the data warehouse.

From the documentation and the samples, I found that it is very easy to INSERT data in a table, but could not find references to table UPDATES, or more specifically to the situation where an input record may be updating the corresponding record in the target table if it already exists, otherwise it will be inserted on the target table.

What would be the best approach to resolve this "UPSERT" type of problem using Openadaptor?

Many thanks for the help

Ismael

@ThingsExpo Stories
DXWorldEXPO LLC announced today that All in Mobile, a mobile app development company from Poland, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. All In Mobile is a mobile app development company from Poland. Since 2014, they maintain passion for developing mobile applications for enterprises and startups worldwide.
Digital Transformation is much more than a buzzword. The radical shift to digital mechanisms for almost every process is evident across all industries and verticals. This is often especially true in financial services, where the legacy environment is many times unable to keep up with the rapidly shifting demands of the consumer. The constant pressure to provide complete, omnichannel delivery of customer-facing solutions to meet both regulatory and customer demands is putting enormous pressure on...
The best way to leverage your CloudEXPO | DXWorldEXPO presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering CloudEXPO | DXWorldEXPO will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at CloudEXPO. Product announcements during our show provide your company with the most reach through our targeted audienc...
IoT is at the core or many Digital Transformation initiatives with the goal of re-inventing a company's business model. We all agree that collecting relevant IoT data will result in massive amounts of data needing to be stored. However, with the rapid development of IoT devices and ongoing business model transformation, we are not able to predict the volume and growth of IoT data. And with the lack of IoT history, traditional methods of IT and infrastructure planning based on the past do not app...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
More and more brands have jumped on the IoT bandwagon. We have an excess of wearables – activity trackers, smartwatches, smart glasses and sneakers, and more that track seemingly endless datapoints. However, most consumers have no idea what “IoT” means. Creating more wearables that track data shouldn't be the aim of brands; delivering meaningful, tangible relevance to their users should be. We're in a period in which the IoT pendulum is still swinging. Initially, it swung toward "smart for smart...
DXWorldEXPO LLC, the producer of the world's most influential technology conferences and trade shows has announced the 22nd International CloudEXPO | DXWorldEXPO "Early Bird Registration" is now open. Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)
As data explodes in quantity, importance and from new sources, the need for managing and protecting data residing across physical, virtual, and cloud environments grow with it. Managing data includes protecting it, indexing and classifying it for true, long-term management, compliance and E-Discovery. Commvault can ensure this with a single pane of glass solution – whether in a private cloud, a Service Provider delivered public cloud or a hybrid cloud environment – across the heterogeneous enter...
The Jevons Paradox suggests that when technological advances increase efficiency of a resource, it results in an overall increase in consumption. Writing on the increased use of coal as a result of technological improvements, 19th-century economist William Stanley Jevons found that these improvements led to the development of new ways to utilize coal. In his session at 19th Cloud Expo, Mark Thiele, Chief Strategy Officer for Apcera, compared the Jevons Paradox to modern-day enterprise IT, examin...
DXWorldEXPO LLC announced today that ICC-USA, a computer systems integrator and server manufacturing company focused on developing products and product appliances, will exhibit at the 22nd International CloudEXPO | DXWorldEXPO. DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of ...
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
I think DevOps is now a rambunctious teenager - it's starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interopera...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...