|By Dave Chappell||
|May 23, 2003 12:00 AM EDT||
Web services have given newfound importance to service-oriented architectures and promise to drive down the cost of integration by providing a standards-based approach to interoperability between applications. The trouble is, what people really want is a new way of doing integration. Until now, we haven't really had a way to incorporate Web services into a meaningful architecture for integrating applications and services into a fabric that spans the extended enterprise in a large-scale fashion. With the advent of the enterprise service bus we have that architecture.
The Java Business Integration (JBI) specification, JSR 208, has set out to define a loosely coupled integration model that aligns with Web services-style distributed computing. The JBI expert group hopes to utilize existing J2EE integration components such as the Java Message Service (JMS), J2EE Connector Architecture (JCA), and the J2EE 1.4 Web services APIs, and add a host of new capabilities as well. The charter of the expert group promises to define an open-ended, pluggable architecture that will support both industry-standard integration components as well as proprietary elements through the use of standardized interfaces.
Coincidentally, the ESB addresses similar goals. It is based on today's established standards, and has real implementations that have been shipping for at least a year. JBI can learn a great deal from what the ESB approach offers.
The Enterprise Service Bus, an Industry Phenomenon
The enterprise service bus (ESB) is a rapidly emerging technology category that is gaining industry attention from analyst firms and the vendor community alike. More than just a new architecture for integration, the ESB promises to change the economics of integration projects by bringing the fabric of integration into the hands of the everyday IT professional. Gartner Group predicts that an ESB infrastructure will be running in the majority of enterprises by 2005 (DF-18-7304). IDC likens the advent of the ESB to the invention of the steam engine or the telegraph. In IDC's report (#29132), Sally Hudson went on to say, "The ESB is an open standards-based technology concept that will revolutionize IT and enable flexible and scalable distributed computing for generations to come. This flexible, reliable infrastructure can enable a new type of corporation - one that is not held captive to rigid IT capabilities, but rather can respond almost instantaneously to meet business opportunities."
Infrastructure vendors are "getting on the bus" at a regular pace now, with early pioneers including Sonic Software, and early adopters including SeeBeyond, IONA, SpiritSoft, Kenamea, and Cape Clear. The ESB has received front-page attention several times from industry weeklies such as InfoWorld and IT-Week. ESB has also been written about in Web Services Journal and WebServices.org, and debated on the theServerSide.com. An employee of a very large ERP vendor stood up in one of my presentations recently and announced to the audience that they are currently in the process of "ESB-enabling" their ERP suite.
The ESB is an architectural concept that is already being deployed in real production environments. Customers like ABNA, a subsidiary of $1B Associated British Foods, have adopted the ESB to provide standards-based integration between its internal systems and electronic trading partners, utilizing the ESB infrastructure for XML document exchange between its supply management systems and ABF subsidiaries, as well as other trading partners.
Changing the Economics Through Standards-Based Integration
Integration brokers have long suffered from being highly proprietary in nature. This proprietary nature has been very problematic to organizations trying to integrate on a large scale because it leads to unwanted vendor lock-in, and usually carries with it a high cost in consulting fees throughout the project. In the end, what is usually accomplished is an expensive island of integration that can essentially only be viewed as an ongoing "integration tax" to be written off as a cost of doing business.
In contrast, the introduction of standards throughout an integration fabric - which in part defines the ESB approach - offers business benefits through the following characteristics:
Characteristics of the ESB Employ Key Standards for Integration
There are a significant number of standards, all of which have reached a fair enough maturity level in implementation that fit together very nicely in an ESB architecture. It could even be posited that the evolution of the standards have helped to catalyze the creation of the ESB. Table 1 shows a sampling of the main characteristics that make up the ESB, and the standards that support those traits.
(Note: In Table 1 the items highlighted with a *, such as WS-Reliability and WS-Security, are not yet part of the ESB since they have not yet reached a sufficient level of maturity and widespread adoption. However they are considered strong future candidates." The ESB is all about standards-based integration based on what is available today, in an architecture that is well suited to support the maturing technologies of tomorrow.)
Getting on the Bus
A common interpretation of the term "bus" is the notion of an architecture that one simply plugs into. Think of a hardware bus in a PC motherboard, with multiple slots that any type of compatible card can be plugged into. The use of the term "bus" in the ESB case is analogous to that in the way that one interfaces with the bus. From the perspective of the owner of a service or an application domain, you need only be concerned with three operations - you plug into the bus, you post data to the bus, and you receive data from the bus. The Enterprise Service Bus takes care of getting the data to the applications that need it, in the target data formats that they need to see it in.
Expanding the Reach of Integration Services
Roy Schulte of Gartner Group summarizes the ESB as "combining Message Oriented Middleware (MOM), Web Services, and Transformation and Routing Intelligence." These are concepts that are found in many traditional integration brokers. The ESB concept incorporates those same concepts, yet applies standards to every aspect possible, and does it in a way that enables a highly distributed, yet centrally managed SOA. Think of the contrast between conventional integration broker stacks versus the ESB approach as centralized and proprietary versus highly distributed and standards based.
An ESB is based on a distributed, federated network model. It takes on the responsibility of reliable communication within the network with a common security model, and provides the integration capabilities that allow applications to easily interoperate. Integration capabilities such as data transformation are themselves implemented as services and can be independently deployed anywhere within the network. The result is precise deployment of integration capabilities at specific locations, which can then be scaled independently as required. It also means that services can easily be upgraded, moved, or replaced without affecting applications.
Intelligent routing based on content is accomplished through specialized services that apply XPATH expressions to identify a document type and route it based on values within the document. Routing of messages is also accomplished using a message itinerary. Think of a message itinerary as a list of destinations that travels with the message as it moves from service to service, or application to application. The ESB evaluates the itinerary at each step along the way, without the requirement of a centralized rules engine. More complex orchestrations, including stateful, long-duration conversations, are also possible using orchestration facilities. An ESB architecture is well suited for supporting choreography languages such as BPSS or BPEL4WS.
In your role as IT's new integration architect, your job is to tie applications and services together in a service-oriented architecture using itinerary-based routing, content-based routing, and orchestration. The interface through which this is done is the ESB services container.
Highly Distributed Services Container
Perhaps the most important characteristic of the ESB is the highly distributed services container, which allows the selective deployment of discreet services exactly when and where you need them. The services container is the interface between the bus and the applications and services that communicate through the bus (see Figure 1).
The services container can house an application adapter via JCA, or it may expose an MDB or JMS interface to talk to a J2EE app server. It may house an XSLT transformation engine that converts a particular XML dialect into an agreed-upon centralized common format, or act as a content-based routing service. A service container may be an XML caching, staging, and aggregation service that gathers data from multiple sources prior to rendering them together using an XSLT transformation, in preparation for feeding into an ERP system. A service container may also act as an interface to external Web service invocations, both inbound and outbound. Combined with the staging service, it could be used as a portal cache for a portal server based on J2EE.
XML transformations can be very CPU intensive. As indicated by Figure 1, services containers can be independently scaled across as many machines as necessary to support the load required.
HTTP and SOAP are no problem for the ESB, as it is capable of exposing SOAP-based Web services to the outside world, and also capable of calling out to external Web services across the Internet. The ESB takes care of mapping SOAP-over-HTTP to SOAP-over-JMS for both inbound and outbound Web services invocations. Emerging WS-Reliabl* SOAP specifications, such as WS-Reliability and WS-ReliableMessaging, can be easily adapted into the ESB architecture as they become widely adopted.
Reconstructing the J2EE Stack
The current approach taken by integration brokers that are built on app server platforms is to install the entire app server stack in every place that a particular piece of functionality is needed. This is referred to as the "AppServers Everywhere" strategy (see Figure 2).
The "Appservers Everywhere" strategy can be viewed as overkill in some cases. Think of installing an entire integration broker or app server stack in a particular department, or remote location, simply to act as a conduit to a JCA container which houses an adapter to an ERP system. Why do that when you can simply place a services container there, and have it act as the JCA container directly?
Don't get me wrong. App servers are good, and have their place in the ESB, for things like managing Web-facing portals and housing EJBs. Through JMS and MDB, all app servers, such as WebLogic, WebSphere, and JBoss, can hook directly into the bus and become a part of its service-oriented architecture. However, the integration functionality is delegated to the ESB itself.
The requirements of integration based on standards are vastly different from the application server view of the world. Today's integration projects require a substrate that is built from the ground up to enable loosely coupled, document-centric XML interchange. This does not mean that the entire J2EE stack needs to be discarded. The ESB is built upon Java standards such as JMS, J2EE Connector Architecture, EJB, MDB, WSEI, JAX-RPC, SAAJ, Java servlets, JMX, JAXP, JAAS, and JSSE. It simply means that a radical rearranging, or "Reconstructing," is in order.
The services containers are connected together through a messaging backbone using JMS. As illustrated in Figure 3, the JMS backbone can support scalable clustering across a large scale global deployment.
The services container is both a JMS client and a JMX MBean container that is capable of housing integration services (see Figure 4). The container receives JMX management input, and has special output endpoints for error handling and auditing.
The services container uses a Java servlet-like inbox/outbox processing metaphor. The destinations that the endpoints eventually map to are administratively defined. Underneath the covers, the in/out endpoints are mapped to JMS topics and queues that carry SOAP envelopes. The topic or queue can be directed at an MDB or a JCA container, or could be mapped administratively to call out to an external Web service using a mapping to SOAP over HTTP, or even WS-Reliabl*.
The service API can be implemented using an open-source SOAP stack such as Apache Soap or Apache Axis, and could conform to Java standards such as JMS, JAX-RPC, or SAAJ. It could be a combination of those. That is probably something that should be standardized by the JBI/JSR208 EG. There is already some API work being done by Sonic and IBM in Apache Axis to build an asynchronous callback API for SOAP services based on combining the JAX-RPC with JMS. The goal was to not reinvent the wheel and to take advantage of existing standards. Perhaps we can draw from this work and use this approach in JBI/ JSR208.
Technology Catalysts Driving Business Models
Mature standards such as the Java Message Service (JMS) for messaging, and the J2EE Connector Architecture (JCA) mean that IT can now pick and choose from best-of-breed implementations and combine them together. Proprietary application adapters are also becoming standardized. Companies such as iWay are selling ESB-enabled adapters, in an a la carte model, that are based on standard interfaces such as JMS and JCA. This is creating a highly competitive market that competes on price, performance, scalability, and functionality. De-facto standard SOAP stacks such as Apache Soap and Apache Axis are licensed freely, yet have strong backing from industry vendors who help build these SOAP stacks for use within their own commercial offerings, and offer their contributions back into the open source community.
According to Gartner Group, an essential characteristic of the ESB approach is "high function, low cost." Low cost is certainly reflected in the licensing, but more importantly in the overall cost of configuration, deployment, and cost of ownership over time. Contemporary integration broker vendors have built their businesses by charging initial license fees ranging from $500K-$1M, and consulting fees typically five times the license cost. In contrast, the licensing fees for an ESB approach can range from one tenth to one half of that cost, with consulting fees an average one tenth of the cost of conventional integration consulting fees. The competitive landscape being created by up-and-coming ESB companies, combined with the business benefits of standards-based integration stated earlier, will take a heavy hit on the business model of the traditional integration broker and the app server vendor alike. Even if the existing vendors can somehow re-architect their products to create a low-cost, highly distributed ESB offering, it's questionable whether their business model can be adapted to survive the new economics of integration.
As ESBs become more widespread, system integrators may adapt to the lower cost/consulting ratio by adopting the ESB approach and taking advantage of its proliferation.
And one last point: although this article focuses on J2EE, the ESB is platform agnostic. A good ESB should be capable of supporting direct access through .NET clients and C++ clients.
I'm very excited about the formation of a JSR in the area of business integration, as is Sonic, because we feel that today's integration requirements call for a vastly different approach from the app server model that was once built for processing Web requests. The ESB approach, along with JSR 208, will "change the economics of integration" and breath some fresh ideas into the J2EE stack.
- ESB Myth Busters: 10 Enterprise Service Bus Myths Debunked
- ESB Integration Patterns
- Universal Middleware: What's Happening With OSGi and Why You Should Care
- Guaranteed Messaging With JMS
- Benchmarking JMS-Based E-Business Messaging Providers
- Distributed Logging Using The JMS
- Reconstructing J2EE-Java Business Integration Meets the Enterprise Service Bus
- Service-Oriented Integration: Making the Right Choices to Support Next-Generation Integration
- The Java Message Service
- A Real-World Example