From the vice president and chief technologist for SOA at Oracle Corporation

Dave Chappell

Subscribe to Dave Chappell: eMailAlertsEmail Alerts
Get Dave Chappell: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: XML Magazine

XML: Article

A Real-World Example

A Real-World Example

Last month "The JavaMessage Service and XSLT for E-Business Messaging" (XML-J, Vol. 2, issue 2) explored the concept of using JMS as the basis of a communications architecture for transporting XML data between applications and an XSLT translation engine for transforming business documents from one form of XML to another.

XSLT works great as a data translation strategy if all parties concerned are already speaking some kind of XML dialect. But the real world isn't like that. XML isn't yet ubiquitous, and there's no big switch on the wall you can flip to have every application in every organization suddenly start speaking it. If you want to do business electronically with other organizations, you can't necessarily dictate what data formats and communications protocols they must use. It's likely that your business partners have legacy systems they can't - or won't - adapt to conform to modern standards.

Whether you're a participant in a supply chain through a number of trading hubs, building your own supply-chain hub for a particular vertical industry, or just want to communicate directly with a finite set of business partners, the issues are the same. How do you build a modern distributed infrastructure using a common communications layer and common data formats and still coexist in this diverse environment?

In this article we'll focus on a real-world example of such a situation. BuildNet, Inc., is a leading provider of supply-chain-wide information synchronization for the annual $800-billion U.S. construction industry. BuildNet provides information synchronization through its Internet-based transactions, its widely used enterprise resource planning and project management software systems, its Portal Group, and its Web site, www.BuildNet.com, which delivers electronic catalogs, construction-related applications, business intelligence, and financial services.

The Challenge Faced by BuildNet
Until recently, the BuildNet exchange featured a single connection method, a single transaction standard, and a single set of paths through the exchange. All partners were expected to conform with the connection method and use the same documents in the same format. Since the original hub was based on a third-party EAI product, each partner had the overhead of installing and maintaining that EAI product's proprietary client and infrastructure at their site. Needless to say, not all partners and potential partners were particularly happy with this arrangement.

Many of BuildNet's partners and potential partners had invested in EDI. Having seen the hassles and confusion in implementing the various EDI "standards" for data mapping and connection methodology, they weren't eager to reprogram their systems to yet another connection model and yet another data "standard." To make matters worse, many of the partners had no access to the source code of their back-office systems, so they had no method of developing interfaces that weren't already a part of the standard product.

As part of the strategy to address "openness" to EDI, BuildNet acquired an Atlanta-based company that also had a hub dedicated to the process of connecting distributors and manufacturers through EDI. They accomplished this by mapping whatever data standards existed on one side to whatever data standards existed on the other and became the connection engine and translation engine for both. The manufacturer's side was fairly straightforward, as most manufacturers support some version and format of EDI.

The distributors' side was quite another story. Most could import some type of file (all needed to be in different formats, of course), but on the output side often there was nothing to work with but documents generated via printouts sent to file. In addition, few distributors had any connectivity, and FTP was the only likely possibility for a reasonable amount of time and money. So the staff in Atlanta built an FTP processor involving a "black box" client for the distributor and a Perl parsing system for digging out the nuggets of relevant data from the received printouts.

BuildNet's challenge, of course, was to bring all this capability together in a single, flexible, easily managed system.

The Landscape
Figure 1 shows the original BuildNet topology. Its hub required a client piece at the customer's site for "polling" the hub. Each polling operation involved a "get" and a "put" to send and receive documents. BuildNet had written an additional piece of software to aid clients in working with the client connection piece. This software kept transaction data in an object database and facilitated the connection and security services. The database in the center of the hub was necessary for buffering, as most of BuildNet's partners didn't stay connected full-time. This solution was considered proprietary and inflexible by the trading partners participating in the BuildNet exchange.

The Atlanta hub consisted of several processes connected by various scripts and programs (see Figure 2). It required a proprietary client connector implemented as a "black box." In this case the black box was responsible for performing FTP and security work back and forth with the hub. As appropriate, received documents were handed to a Perl scripting engine that took report formats and turned them into a flat canonical form. These were then passed through a mapping exercise and an EDI processor to be sent to a VAN. Return documents followed roughly the same flow, but were put out to the client in a flat file format, bypassing the Perl scripts. New Perl scripts and maps were developed as needed for each new client.

The Solution
In any new solution, BuildNet wanted:

  • To open up their connectivity options dramatically, to connect to a client via whatever method the client wanted.
  • To offer any-to-any document translation services. The new system shouldn't have to care what format the input came in or what format the output was expected in.
  • An environment in which services could be added to the hub easily and were specific to certain processes and clients.

XML as the Canonical Data Format
The hub itself does more than just consume documents and spit them out again. Many applications and services operating behind the scenes facilitate automation of the supply chain, including scheduling, bid management, collaboration, procurement, logistics, and customer service. Routing and billing applications are prime examples. A request for a price quote arriving at the hub needs to be examined and routed to the indicated suppliers. When a set of quotes is selected, a builder needs to generate a set of purchase orders. Suppliers that provide goods to the builders also need to send an electronic invoice, which is the responsibility of specialized routing and billing applications within the hub.

Since data coming in and out of the hub almost always needs to be translated, it makes sense that all services and applications within the hub use a common format. Thus the outside world can change as much as it likes, and the applications within the hub remain stable. This is where XML is used exclusively. BuildNet had standardized on its own dialect of XML - BuildNetXML - and is in the process of supporting xCBL, the XML standard created and championed by Commerce One (www.xcbl.org).

XSLT can be used as the standard way to generate the output data in formats the receiving trading partner expects to receive. Since XML is the canonical format within the system, it's easy to do that.

As more trading partners move toward supporting XML directly, Perl scripts and EDI mappers will become less important. XSLT will become the predominant choice for data translation for both inbound and outbound pieces. However, that may be years away. Even then it will be important to be able to support an architecture that allows multiple forms of translation engines to be easily plugged into the system.

The New BuildNet JMS Hub
Just as it makes sense to conform to a canonical data format within the architecture of the hub, it made sense to conform to a common communications architecture within the hub. BuildNet has transitioned completely to a JMS-based hub. As part of this transition, they've implemented a number of changes in approach and architecture. The use of JMS at the center of the architecture provides the following benefits:

  • A standardized API for all applications and services within the hub.
  • Asynchronous processing of all business transactions throughout the supply chain: Through JMS, senders and receivers are abstractly decoupled from one another.
  • Guaranteed message delivery semantics of JMS: With other forms of nonguaranteed protocols, higher-level acknowledgement of data receipt had to be built in at the application layer. JMS provides guaranteed once-and-only-once delivery of data, which removes that complexity from the application developer.
  • The publish and subscribe messaging model, which allows the one-to-many broadcast of information: Through the pub/sub model, monitoring and auditing services within the hub can be selectively added at a fine-grained level simply by having the services subscribe to certain pub/sub topics.
  • A service-based architecture: Using JMS, new services may be added to the system dynamically simply by adding applications that communicate through topics and queues. Data transmissions may be intercepted and rerouted without affecting every application within the system.

The complexities of the communications infrastructure are dealt with by the JMS vendor, allowing BuildNet engineers to focus on the higher-level architecture of the system and supporting applications and services that perform the business functions within the hub.

In addition, having JMS at the heart of the architecture allows a single, simple approach to the multiple-communication protocol issue. In the new architecture protocol, bridges are merely specialized JMS clients that act as gateways to the outside world (see Figure 3). These bridges act as proxies between the various protocols and JMS.

After being accepted through a protocol bridge, the first thing any transaction goes through is a translation engine, which takes the document and puts it into the standard canonical form. For example, an SMTP bridge may accept an incoming e-mail transmission and wrap it up inside a JMS message envelope. The JMS message is then placed into a queue for processing. A routing service listening on that queue then forwards the message to the appropriate translation service based on properties set in the message header. Even for those established trading partners who may be sending an XML document based on an older version of BuildNetXML, it's a simple choice of forwarding to an XSLT translator. Anything downstream from this is then in a known good form, so any other service subscribing to the topics can handle the data regardless of source.

For backwards compatability with existing trading partners, they've implemented a bridge that knows how to talk to the EAI client that may still be installed at a trading partner site. They've even implemented a bridge that allows an application at the trading partner site to communicate with the hub using SQL statements through a JDBC interface. Let's not forget, of course, the pure JMS client that allows a trading partner to use JMS directly.

Once processed by the applications within the hub, the document is sent to the document transformation services once again - its final step out the door. This type of door is controlled by entries in a directory service that are maintained for each trading partner in the exchange. These entries indicate the connection method desired on the outbound side, the data format, and other rules of engagement.

There's no longer a need to have a separately maintained hub with the sole purpose of performing EDI conversions. With this new JMS-based architecture, the translation portion of the Atlanta EDI hub can simply be added into the main BuildNet hub as just another translation service.

Conclusion
When planning an open communication with multiple outside entities, whether building your own portal or communicating with other portals, or directly with your business partners, it's important to consider these issues:

  • Open connectivity methods: You can't play unless you adapt to the user's method of communicating.
  • Open document formats: You can't win by asking everyone to change to your format. You need to accommodate everybody. XML and XSLT are great, but they're not everywhere yet.
  • Central canonical form: Once it's in your system, you need to have any service be able to use it. XML is ideal for that.
  • JavaMessage Service: The loosely coupled nature of a JMS system is really the only answer to doing this kind of work in a flexible environment. You want to be able to bring new services on and alter existing services with little disruption.

More Stories By Dave Chappell

David Chappell is vice president and chief technologist for SOA at Oracle Corporation, and is driving the vision for Oracle’s SOA on App Grid initiative.

More Stories By Neil Powers

Neil Powers is CTO for the e-commerce division of BuildNet. He has worked in the computer industry for 20 years, 16 with BuildNet and their subsidiary, NxTrend Technology. An adjunct professor at the University of Phoenix, Neil holds a master's degree in computer information systems and a bachelor's degree
in English.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.