axis-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chad La Joie <>
Subject Re: End-to-End Streaming Support
Date Sun, 05 Aug 2007 20:15:17 GMT
Okay, so it does seems that I haven't explained what I need clear enough.

I'm implementing a SOAP service based on a particular specification.
One operation of this service allows for a client to upload arbitrarily
large binary objects, as base64 encoded content of an element, within
the SOAP body.  The specification then requires the use MTOM as a means
of optimizing the transfer of this information.  The SOAP service has
another operation that allows for a client to retrieve this uploaded
information in the same manner (base64 encoded content of SOAP body
element using MTOM during transmission).

The binary content being uploaded could be something small (1-2MB) or
could be quite large (many GBs).  In the later case, if, at any point
during transmission, on either the client or server side, some bit of
code tries to buffer, in memory, the whole message or even just the
base64 encoded content the process will exhaust the server's memory.
So, what I need is true end-to-end IO streaming support.

By this I mean I need the client to be able to begin building and
sending the XML over to the service.  When the client gets to the point
where it needs to send the contents of the element carrying the base64
encoded binary content, it would pipe the DataHandler's input stream
into the established output stream to the service.

Likewise, on the service side, it would receive the first parts of the
SOAP request (the headers and the start of the body) and when it got to
the point where it was receiving the base64 encoded binary content,
would pipe its input stream into the output stream of the DataHandler.

Reverse this for the second operation where the client is pulling the
information down from the service.

I then asked, if Axis2 did have such support, was it possible to, on the
service side, for the data handler to receive the information in base64
encode form during the first operation and provide it in base64 encoded
form during the second operation.  The service itself never operates on
this encoded binary content (except to store it).  So, if it could
receive, store, and provide it all in the base64 form that would be
significantly more efficient.

So, in summary, the key point is that neither the client or service ever
make the assumption that it can fit any MIME body of the XOP package in
memory, other than the initial aplication/xop+xml body, in memory.

Martin Gainty wrote:
> Chad-
> By streaming your definition is not necessarily read or write (XML |
> Text) streams but something closer to RSS Feed I would look at tweaking
> axis2/databinding/utils/reader/ specifically
> take a look at newDocument method
> notice the wrap of ByteArrayInputStream("SomePreviouslyReadByteArray");
> and use that for your InputStream
> you'll have to think thru how much to read (is this entity RLE?)
> (Please confirm whether your stream is an RSS or using some other
> Streaming mechanism)
> HTH/
> Martin
> This email message and any files transmitted with it contain confidential
> information intended only for the person(s) to whom this email message is
> addressed.  If you have received this email message in error, please notify
> the sender immediately by telephone or email and destroy the original
> message without making a copy.  Thank you.
> ----- Original Message ----- From: "Chad La Joie" <>
> To: <>
> Sent: Sunday, August 05, 2007 1:08 PM
> Subject: Re: End-to-End Streaming Support
>> Thilina Gunarathne wrote:
>>>> Does Axis, or Axiom, handle the Base64 encoding/decoding of the binary
>>>> data to/from a DataHandler?
>>> Yes... Only if needed...
>>>> If so, is there a way to disable such handling?
>>> Nope... But if you use the API properly you can avoid any unncessary
>>> encoding/decodings.
>>>> Characteristics for my service have a client submitting large files and
>>>> then some number of clients retrieving them.  The service itself treats
>>>> these files as opaque blobs.  It could receive, and return, them to
>>>> Axis
>>>> in a Base64 encoded form saving a lot of time when clients retrieved a
>>>> document be avoiding a re-encode.
>>> Above is not clear to me. Is your service written in Axis2. Do you
>>> want to send/receive binary in base64 form.. If so it won't tolerate
>>> large files..
>> I'm not using anything yet.  My hope was to use Axis2 so right now I'm
>> just trying to make sure it supports what I'll need.  I know that the
>> need to send large files, encoded as base64 element content, is not
>> something that most SOAP stacks handle.  This is why I started by asking
>> about true end-to-end streaming support when using MTOM.
>> If, as we discussed in the first part of this thread, Axis2/Axiom
>> supports true end-to-end streaming, and Axiom can correctly stream
>> content from/into a DataHandler, what causes problems for arbitrarily
>> large files?
>> Maybe I misunderstood something in the first part of the thread or
>> didn't properly state what I was trying to accomplish.
>> -- 
>> Chad La Joie             2052-C Harris Bldg
>> OIS-Middleware           202.687.0124
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Chad La Joie             2052-C Harris Bldg
OIS-Middleware           202.687.0124

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message