Managing transmittal of large amounts of data via web services.

An interesting situation has come up at work where it may be necessary to transmit large amounts of data between applications via web-services.  Ideally, all this data could be processed and filtered in the application that generates it but this may not be possible in the future.

Luckily, we just ordered Enterprise Integration Patterns Designing, Building, and Deploying Messaging Solutions.  This book is programming language agnostic and reads like a cookbook of best practice solutions.  Having referred to the book but a few times, I think this book is well worth having close by when developing a service oriented architecture.

A sensible solution to this situation, according to the book, is to break the big message into smaller messages. The general idea being that the big message can be broken down into send-able messages.  For the situation above described, we would be breaking down big message into n number of messages and adding to each three additional pieces of information.  I am specifically referring (and quoting) from ‘Message Sequence’ from book as a possible solution to situation.

First, we would include in each smaller message a ‘sequence identifier’ that would identify all the messages as part of a bigger set (main message).  Secondly, we would include a ‘position identifier’ stating which message in the series a particular message is.  Lastly, a ‘size’ or ‘end indicator’ would be sent, which will inform either the number of messages in the cluster or would identify the last message in the cluster of messages. I believe that sending number of total messages in cluster is best in this situation.

For example.  Lets say we have a web service which generates 600 kbs of data that we want to send to a distributed application far away.  Furthermore, lets state that the recipient application can only consume or process a limited amount of data at a time.  We may want to break down message into, say, 50 kb chucks.  This means that, instead of sending one big xml packet containing 600kbs of data, we would send 12 smaller xml packets containing 50 kbs of data each.

From the previous description of the modifications required to messages, we would obtain xml packets like this for example:

  <sequence>(UID identifying message set about to transmit)</sequence>
  <position>(Identifies message from set of messages to be sent)</position>
  <size>(This is the number of messages about to be sent)</size>
  <data>(50 kbs data chunks)</data>

The recipient application now has all the information it needs to reassemble the big message for processing.  Even, better, the recipient application has enough information to poll sender regarding the big, original message.

For example, if one of the smaller messages gets lost in transmission, the recipient application could request this message be sent again.

I think this is a very elegant solution.  Most often, data passed between applications has been combined and concatenated to such an extent that it needs a separate explanation to be comprehensible.  Simple is best for me. I think this book will be a big time saver since this is but one of the many solutions to the stated problem.  Having these fully explored in book from very reputable sources helps me both learn more about enterprise application development and gently steers me away from possible pitfalls if I fail to consider difficulties I may not be aware of.

I am looking forward to using this book as a resource for both development at work and as instructional material.  Even thou the title sounds a little intimidating if not fluffy, it reads nothing like that.  It reads much more like a textbook for a class you wish you had taken in college but did not exist.  I am very (perhaps, too easily) impressed.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s