From the vice president and chief technologist for SOA at Oracle Corporation

Dave Chappell

Subscribe to Dave Chappell: eMailAlertsEmail Alerts
Get Dave Chappell: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Article

Reliable SOAP for Web Services Messaging Has Finally Arrived! Leading IT Vendors Join Forces to Create Web Services Reliability

Reliable SOAP for Web Services Messaging Has Finally Arrived! Leading IT Vendors Join Forces to Create Web Services Reliability

(January 14, 2003) - On Thursday January 9, Sonic Software and a number of other leading IT vendors, including Fujitsu Limited, Hitachi, Ltd., NEC Corp, Oracle Corp., and Sun Microsystems, announced a proposal for a new Web services specification for reliable messaging: Web Services Reliability (WS-Reliability). The companies plan to submit WS-Reliability to a standards body on a royalty-free basis in the near future.

Along with security, reliable asynchronous communications has been one of the gaping holes in today's Web services architecture. Lack of reliability, due to the inherent nature of using SOAP over protocols such as HTTP, is one of the biggest obstacles to the adoption of Web services for mission-critical communications between applications and services, such as complex business-to-business transactions or real-time enterprise integration.

WS-Reliability is a standalone specification for reliable SOAP that can provide the missing link to bridge the gap between organizations and help make Web services a truly enterprise-capable technology for standards-based systems integration. As one of the WS-Reliability specification authors, I thought I would drop a quick note and give a description of it, and my viewpoint on where it is headed.

SOAP-Based Reliability

WS-Reliability is a specification for an open, reliable SOAP-based asynchronous messaging protocol, which enables reliable communication between Web services. WS-Reliability includes well-known characteristics of reliable messaging such as guaranteed delivery, duplicate message elimination, and message ordering. Delivery options such as at-least-once, at-most-once, and exactly-once are available. Message expiration and delivery retries are also possible.

I use the term SOAP-based reliability to mean that the reliability capabilities are defined in the SOAP envelope. The spec defines a set of SOAP envelope headers that govern the behavior of the senders and the receivers. This means that even while an HTTP binding is provided, WS-Reliability is a level above, or independent of, the underlying protocol.

Listing 1 shows a partial listing of a WS-Reliability SOAP header.

<SOAP:Envelope
?
<SOAP:Header>
<rm:MessageHeader xmlns:rm="http://schemas.fujitsu.com/rm"
SOAP:mustUnderstand="1">
?
<rm:MessageId>20020907-12-34@anyuri.com</rm:MessageId>
<rm:Timestamp>2002-09-07T10:19:07</rm:Timestamp>
</rm:MessageHeader>
<rm:ReliableMessage xmlns:rm="http://schemas.fujitsu.com/rm"
SOAP:mustUnderstand="1">
<rm:ReplyTo>http://server1.anyuri.com/service/</rm:ReplyTo>
<rm:TimeToLive>2002-09-14T10:19:00</rm:TimeToLive>
<rm:AckRequested SOAP:mustUnderstand="1" synchronous="false" />
<rm:DuplicateElimination/>
?
</rm:ReliableMessage>
?
</SOAP:Header>
?
</SOAP:Envelope>

Listing 1: A sample WS-Reliability SOAP Header

Guaranteed Delivery

The guaranteed delivery is accomplished using concepts such as message persistence, acknowledgement of receipt, and error reporting via SOAP Fault handling. If a message is tagged as being a <ReliableMessage> with a <AckRequested> element in the header, this is a signal to the ultimate receiver that it must send back an acknowledgement message to the sender--or a Fault message if something fails. While WS-Reliability is an asynchronous protocol, the acknowledgement or Fault may be sent either synchronously or asynchronously. Acknowledgements and Faults are correlated with the originally sent message using the <RefToMessageId> element.

A full section on Fault handling, including a substantial list of Fault codes, and some sample Fault envelopes are provided.

Duplicate Elimination

The duplicate elimination is accomplished in part by the use of globally unique message identifiers in accordance with [RFC2822] http://www.ietf.org/rfc/rfc2822.txt. In case you're wondering, they look like this: <rm:MessageId>20020907-047184-0297@someuri.com</rm:MessageId>. It is the responsibility of the receiving Reliable Messaging Processor (RMP) to detect duplicates and only deliver one message to the receiving application, discarding the rest. Now, you may notice I didn't say keep the first and discard all subsequent messages. We left it open for now because who is to say whether the first message is to be kept, or the last one. That's an example of an application specific requirement and something that perhaps could be configurable in an implementation specific RMP.Message Ordering Message ordering is accomplished using a <MessageOrder> element, which contains both a <GroupId> and a <SequenceNumber> element. The receiving RMP is responsible detecting out of sequence messages, and either waiting for them to arrive, or generating a Fault back to the sender.

Message Persistence

Both the sending and receiving RMPs are required to persist the messages under certain conditions. The sending RMP is required to persist a message until it receives an acknowledgement message, the time span as indicated by the <TimeToLive> element has expired, or a configurable number of retry attempts have failed. The receiving RMP is required to persist a message until a receipt has been delivered back to the sender, and all ordering and duplicate elimination requirements have been satisfied.

Say It Isn't So!

Many of you may know me as an active proponent of JMS reliable message delivery semantics, and you may be asking "What's this all about"? Well, the short answer is that this is very complementary technology. Reliable messaging comes in many forms, and is a key part of Sonic's core competencies. WS-Reliability should become a natural fit into any environment that supports JMS, HTTP, SOAP, and Web services.

Where the Specification is Headed

The intent of this draft is to act as the basis for input into the formation of a working group or technical committee in a standards body. We are in the process of working through that right now. I don't want to publicly say which organization until it actually happens, but stay tuned. We welcome the addition of any and all companies that wish to join in with us once we get it to a standards body. We're already being contacted by numerous companies who are excited about joining in.

The intent of WS-Reliability is to provide a simple yet robust specification for reliable messaging that can work well within existing parts of the Web services stack, and be capable of fitting in with other complementary efforts as they progress.

We realize that the specification is in its early stages and we still have a great deal of work ahead of us, and look forward to the input from the other companies as they come on board with the effort. I encourage you to go download the specification and have a read through it. It is pretty easy reading, and has plenty of interaction diagrams and many more sample SOAP envelopes to help you understand these concepts in more detail.

At the moment the specification is published at each of the participating vendors' respective Web sites. You can get a current copy of it at http://www.sonicsoftware.com/wsreliability. A full list of locations is provided at the end of this article.

By providing an open standard for a SOAP-based reliable transport, WS-Reliability will help accelerate adoption of asynchronous Web services, making them relevant for an even wider range of standards-based integration across the extended enterprise, and cross-company collaboration challenges.

Alphabetical list of sites to download the WS-Reliability specification from:

Fujitsu
http://xml.fujitsu.com/en/about/ws_r.html

Hitachi
http://www.hitachi.co.jp/soft/wsrm/

NEC
http://www.nec.co.jp/press/ja/0301/WS-ReliabilityV1.0Public.zip

Oracle
http://otn.oracle.com/tech/webservices/htdocs/spec/ws-reliability.html

Sonic Software
http://www.sonicsoftware.com/wsreliability

Sun Microsystems
http://sunonedev.sun.com/platform/technologies/technologies_mv.html

More Stories By Dave Chappell

David Chappell is vice president and chief technologist for SOA at Oracle Corporation, and is driving the vision for Oracle’s SOA on App Grid initiative.

Comments (3) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Dave Chappell 02/24/03 07:49:00 PM EST

Hi Peter,
Thanks for bringing these postings to my attention. I just responded directly to those two blogs, but here it is again -

Issues from http://radio.weblogs.com/0108971/2003/01/15.html -

First, let me say thank you for pointing out these issues. Many of these things were discussed during the initial formation of the spec, and we decided that it would be better to wait until the formation of the OASIS TC, which has now happened. Although the initial spec was posted as a group of vendors collaborating on a specification, our goal all along has been to use the initial spec as input to the formation of a WG or TC. We didn't want to go down too many implementation detail paths, particularly when it comes to things like inherent requirements on the underlying infrastructure. We also didn't want to go too far down a
"proprietary" path as a rogue gang of vendors, without bringing it to a broader forum such as an OASIS TC. We look forward to ironing these types of issues out with the other WS-RM TC members. That being said, let me see if I can address your points specifically -

>Because WS-Reliability is unaware of and not integrated with WS-Routing, it is only useful as
>a point to point mechanism. While routing from the sender to the receiver will likely be
>possible, the "ReplyTo" to send the acknowledgement message to does specify a plain URL and
>doesn't allow integration with a reverse path as per WS-Routing. This means that unless the
>ACK message can be piggybacked on a synchronous response (the luckiest of all circumstances),
>the spec requires either direct connectivity from the receiver back to the sender, which may
>be impossible due to firewalls and NAT, or requires some form of acknowledgement message
>dispatcher gateway at the sender's site, which requires some form of central service
>deployment as well. In short: This doesn't really work for a desktop PC wishing to reliably
>deliver a message to an external service from within the corporate firewall. "

Good issue. We actually had many discussions and early versions of the spec that had attempted to address multi-hop, and perhaps even WS-Routing. Multi-hop issues in general are
being discussed in other work groups like XMLP (SOAP 1.2), WS-Architecture, and WS-I. We look forward to converging with those discussion to make sure we are in step and doing the right thing. There is also a bigger issue with WS-Routing in particular in that it is thus far a proprietary specification.

Another point is that the growing trend in the industry for supporting asynchronous messaging-style web services communication for interactions within and across the extended
enterprise is going to mean that most organizations will host asynchronous listeners anyhow.

WS-Reliability is not driving the charge there, its already happening. I agree that there still needs to be some sort of routing or dispatching necessary to get back to the desktop PC.

hat's a good issue to flesh out in the TC.

>There's quite a few problems to be solved with regards to simple sequence numbers and resends

>of an unaltered, carbon-copy (2.2.2) of the original message considering the accuracy of

>message timestamps, digital signatures, context coordination and techniques to avoid replay

>attacks. Sending the exact same message may be entirely impossible, even if it couldn't be

>delivered properly and therefore the "MUST" requirement of 2.2.2 cannot be fulfilled. Also,

>in >2.2.2 there's a reference to a "specified number of resend attempts" -- who specifies

them? "

We chose to use the message id as the thing that determines whether a message is a duplicate, for these reasons. The specified number of resend attempts is intended to be a configurable option, but falls under the category of a requirement on the underlying infrastructure, which is yet to be specified.

>The spec rightfully calls for persistent storage of messages (2.2.3), but doesn't spell out
>rules for when messages must be written to persistent storage in the process (it should
>obviously before sending and after receiving, but before acknowledgement and forward).

I thought that section 2.2.3 was pretty clear about it. I will make a note of that as an item of discussion in the TC.

>What I find also very noteworthy is that the authors say that they have yet to address
>synchronization between sender and receiver and establishing a common understanding by sender
>and receiver about whether the message was properly delivered (meaning that the send/ack
>cycle >was fully completed). I assume that once they do so, they'll throw the synchronous,
>piggybacked reply on top of HTTP out of the window, because this creates an in-doubt
>situation >for the acknowledging party.

That situation is currently addressed by message redelivery on the sender side, and dupe elimination on the receiver side. We will make a note to revisit this in the TC discussions.

Now that we have formed an OASIS TC, you have a public place to have these discussions. Feel free to post your feedback to wsrm-comments@oasis-open.org.

Issues from http://weblogs.cs.cornell.edu/AllThingsDistributed/archives/000013.html

>The requirement that messages need to be persisted has not been thought through well enough
>(as Clemens already hinted at). The operation on the sender side seems obvious, when you
>recover you try to get acknowledgements for those message you think you have sent, but may
>have gotten lost in the crash. However at the receiver this is less obvious. What does it
>mean to have delivered the message to the application successfully? Can you be sure about the
>point of the possible crash? Can you be sure never to deliver duplicate messages to the
>application during recovery? Does the app also needs to handle duplicates? There are no
>conditions specified for how to remove received messages from the persistent store at the
>receiver.

Issues 3 + 4 in appendix 2 are general statements that we need to further refine the semantics of failure and recovery. Many of us in the TC have very strong experience in Enterprise messaging and are very capable of figuring this stuff out.

>What are exactly the semantics of an acknowledgement? Does this means the message was stored
>in persistent storage? Or that it was successfully delivered to the application?

My view of it is that the message can be considered acknowledgeable once it has been safely persisted. Issues of undelivery to the application can be addressed by the notion of a centralized fault location, or dead message queue, as noted in Appendix 2, section 3.

>What does time-to-live really mean in case of persistent storing your received messages. I
>can send an ack telling the sender I received the message, then I get delayed for some reason
>(maybe a crash) and when I want to deliver the message I notice that its time has expired .
>According to the current spec I cannot deliver this message and have to drop it. Hence the
>message transport becomes unreliable.

Also addressed by Appendix 2, section 3. Look forward to other alternatives which can be discussed in the WSRM OASIS forum.

>The requirement to send a simple ack immediately for each message will introduce a real mess.
>The scenario in which a message gets lost and a subsequent message is received, will trigger
>an ack for this new message making the sender believe that it is reliably received. However
>the receiver cannot deliver the message to the app until it has received the retransmission
>of the missing message. This can cause unreliable behavior because you may have to drop the
>message if there is a ttl field, or if the sender crashes before it could retransmit the
>missing message, the sender gets stuck with the message it has received for ever without
>being able to deliver. The solution here should have been to do a delayed ack or send a
>negative ack, allowing the receiver to treat the new message as volatile until the
>retransmission gap has been filled.

This is recognized by section 6 in Appendix 2.
Dave

Peter Wolf 01/16/03 02:32:00 PM EST

The spec is incomplete, very vague in the description of the techniques, and even errornous at several points. The errors could force the protocol to become unrealiable defeating its whole purpose. For more details on the problems with the spec see the weblogs of Clemens Vasters (http://radio.weblogs.com/0108971/2003/01/15.html) and Werner Vogels (http://weblogs.cs.cornell.edu/AllThingsDistributed/archives/000013.html).

Mark Cinos 01/13/03 07:09:00 PM EST

Reliability delivered with the same level of open standards web services them selves is obviously needed. If the EAI vendors get smart they'll exopse their buses as standards based services, just as some JMS vendors are trying to do (given that there is no interoperability with JMS vendors without proprietary bridges). Today it makes more sense to seperate the JMS layer from the Web Services layer, but recent surveys suggest that 80% of the Web Services will be delivered through either existing platform vendors, or development vendors. That being the case let's hope IBM et al support a standards based platform for reliability.