axis-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "graham glass" <>
Subject RE: Streaming RPC calls
Date Mon, 23 Sep 2002 07:54:25 GMT
hi martin,

glue currently creates a dom representation of a soap message before
funnily enough, when doing benchmarking of soap messages that have an
representation > 1Mbyte, this approach seems to be faster than vendors that
tout a pure streaming approach.

i question whether your requirement of streaming millions of records in a
message is appropriate. is there not a better way in which the client
requests a
subset during a particular operation? alternatively, passing the records as
attachment bypasses this problem completely.

the main drawback that we've found regarding a pure streaming architecture
(we have considered this many times in the past) is the extra complexity it
injects at the API level for those who want to add interceptors, XSLT
transforms, etc.

  -----Original Message-----
  From: Martin Jericho []
  Sent: Monday, September 23, 2002 1:29 AM
  Subject: Re: Streaming RPC calls

  Hi Graham

  Thanks for your interest.  I have included some source files to show you
how I did the streaming.  It's not a complete set so it doesn't compile, but
it should give you an idea how the client side works at least.  (Sorry for
posting an attachment everyone!  It's only 10K)

  The original BatchItemStreamArray class as generated by WSDL2Java simply
contains an array of BatchItem objects called batchItems.  Either the client
or server can use this default implementation, at the risk of running into
memory problems if the array is large.  The new BatchItemStreamArray class
emulates the original class, but extends the StreamArray class, which
contains all the hooks necessary to serialize/deserialize on the fly without
having to actually create the array.  It's all a bit unrefined at the
moment, since I'm just doing a proof of concept as a first step.

  The application I am designing must be able to cope with hundreds of
thousands or even millions of records.  The size of each record is dynamic,
so I can't put an exact figure on the number of MB.

  Does Glue have the same limitations as axis in this regard?

    ----- Original Message -----
    From: graham glass
    Sent: Monday, September 23, 2002 2:38 PM
    Subject: RE: Streaming RPC calls

    hi martin,

    out of interest, how would you write an interceptor to process inputs
and outputs during
    the streaming process? one advantage of having a DOM represention is
that it's easy to
    manipulate during inbound/outbound messaging using tools like xalan.
would you provide
    some kind of streaming interceptor interface?

    also, out of interest, how large is the data you are sending (on the

      -----Original Message-----
      From: Martin Jericho []
      Sent: Sunday, September 22, 2002 11:31 PM
      Subject: Streaming RPC calls

      I have implemented a special class which allows streaming of complex
bean types in an RPC call, but have found that AXIS stores the whole message
in memory before sending / deserializing it anyway, which defeats the whole
purpose of what I am trying to do.

      The first problem is that AXIS doesn't support chunked http messages,
so the content length must be known before sending it.

      QUESTION 1:  Are there any plans to implement chunked transfer
encoding soon?

      The SOAPPart class stores the entire message as a byte array before it
is sent across the wire, which apart from the reason above is unnecessary.

      QUESTION 2:  Are there any plans to clean this up to allow direct
streaming from the serializers onto the wire?

      I have only researched the client side to see how and why the entire
message is stored in memory before sending it.  I have also noticed that the
server stores it all in memory before deserializing it, but I haven't
investigated whether there is any good reason for this.

      QUESTION 3:  If AXIS won't support this any time soon, which I assume
is the case, does anyone know whether the Sun JAXRPC reference
implementation suffers the same problem?  I would also like to know whether
the MS .NET implementation has the same problem if anyone knows about it.

      QUESTION 4:  All I really want to do is send a large array of
structured data in a platform independent way.  I would like to use the
standard RPC encoding of SOAP to avoid having to define my own XML schemas
for the data, but I'm not sure whether today's SOAP implementations are
mature enough to use for this purpose.  What do other people do in this
situation?  I can't imagine I'm the only one.

      I would be more than happy to share my streaming
serialization/deserialization code if anyone is interested, although it is
not of much use with the current version of axis.

      Martin Jericho

View raw message