axis-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Sosnoski <...@sosnoski.com>
Subject Re: Large Web Services responses - how do you deal with them?
Date Tue, 19 May 2009 05:29:17 GMT
Ah, I hadn't seen noticed/remembered this dealt with a single huge 
base64 string. That's just plain bad design for the web service... but 
no surprise if Exchange is involved.

No, you can't handle this cleanly with JiBX or any other data binding 
tool I'm aware of. The only way you *could* handle it would be by 
getting an XMLStreamReader and reading the data directly, with . The 
XMLStreamReader interface provides next() and getTextCharacters() 
methods which in theory could be used to get this text a chunk at a 
time. Assuming the parser involved actually implements these as 
intended, you can get a block of text at a time and run the base64 
decoding on that block, then move on to the next block. A lot of work to 
implement, though.

Andrew, you might want to try switching to ADB or JiBX anyway, at least 
to try for an easier solution than working directly with the 
XMLStreamReader. Either of these should use less memory than XMLBeans 
for the same data, so as long as your text strings are only very large 
and not unbounded these might let you squeak by.

  - Dennis


Andreas Veithen wrote:
> Dennis,
>
> I think that the Web service is actually Microsoft Exchange, so
> changing it is not an option. Does JiBX support caching a base64 value
> (which is not represented using MTOM) on disk instead of memory?
>
> Andrew,
>
> You might still want to give MTOM a try. Many servers switch to MTOM
> if the request is sent as MTOM. I can't believe the Microsoft Exchange
> can only return inline base64.
>
> Regards,
>
> Andreas
>
> On Mon, May 18, 2009 at 13:22, Dennis Sosnoski <dms@sosnoski.com> wrote:
>   
>> Hi Andrew,
>>
>> Your best starting point is to get rid of XMLBeans. XMLBeans always stores
>> raw XML in memory, so there's no way to avoid the memory issues with large
>> messages.
>>
>> ADB would avoid some of the overhead, in that it would convert the XML
>> message to an object graph, which would typically be a factor of 2-5x
>> smaller than the raw XML data (depending on the type of data in your
>> message).
>>
>> JiBX would do at least as well as ADB in terms of the reduced-size object
>> graph. If you have some way of breaking up the response data into more
>> easily-digestible chunks, JiBX would also allow you to do piece-meal
>> processing of the message (by creating a fake collection, for instance,
>> where your data objects expose an addXXX() method which just writes the
>> object being added to some backing store rather than adding it to an
>> in-memory collection). Piece-meal processing is the only way you can
>> dramatically decrease your memory usage and handle messages of effectively
>> unlimited size without a problem.
>>
>>  - Dennis
>>
>> --
>> Dennis M. Sosnoski
>> SOA and Web Services in Java
>> Axis2 Training and Consulting
>> http://www.sosnoski.com - http://www.sosnoski.co.nz
>> Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
>>
>>
>>
>> Andrew Bruno wrote:
>>     
>>> Mr J,
>>>
>>> But its just one data element that I need, so there is no concept many
>>> rows, etc.
>>>
>>> As a matter of fact, I am breaking up the call in 2 calls, one for all the
>>> metadata, and one for the actual mime content.
>>>
>>> i.e.
>>>
>>>  <m:ResponseMessages>
>>>    <m:GetItemResponseMessage ResponseClass="Success">
>>>      <m:ResponseCode>NoError</m:ResponseCode>
>>>      <m:Items>
>>>        <t:Message>
>>>          <t:MimeContent CharacterSet="UTF-8">RnJvbTogTGVlIF..... large
>>> mime base64 encoded email......... 120Meg++ of encoded data</t:MimeContent>
>>> ....
>>>
>>> I only ever request one message at a time.  Its just that when the
>>> MimeContent is greater then 10Meg, OutOfmemory occurs.
>>>
>>> It appears to be the case because the parse loads all the content into
>>> memory.
>>>
>>> In theory, this element can have data of any size coming back.
>>>
>>>
>>>
>>>
>>> On Mon, May 18, 2009 at 6:54 PM, J. Hondius <jhondius@rem.nl
>>> <mailto:jhondius@rem.nl>> wrote:
>>>
>>>    I'd think about a different design of the webservice calls.
>>>    You should try to avoid real big results.
>>>    Split into more calls.
>>>
>>>    Something like:
>>>    One to get a overview list
>>>    Another to get details
>>>
>>>    Or:
>>>    One call to get the size like the SQL count() does
>>>    combined with
>>>    Add parameters to your to limit the result: like start_at,
>>>    number_of_results
>>>
>>>    my 2c
>>>
>>>    Andrew Bruno schreef:
>>>
>>>        Hello all
>>>
>>>        I was wondering how some of you may be dealing with web
>>>        service calls
>>>        that result in extremely large data responses?
>>>
>>>        I have been struggling in trying to change the way the parsing
>>>        of the
>>>        XML response works, as I am getting out of memory errors
>>>
>>>        java.lang.OutOfMemoryError: Java heap space
>>>               at
>>>        org.apache.xmlbeans.impl.store.CharUtil.allocate(CharUtil.java:397)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:506)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:419)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.CharUtil.saveChars(CharUtil.java:489)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:2911)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.stripText(Cur.java:3113)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Cur$CurLoadContext.text(Cur.java:3126)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Locale.loadXMLStreamReader(Locale.java:1154)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:843)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:826)
>>>               at
>>>
>>>  org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:231)
>>>               .....
>>>
>>>        Is there a way to change the parser to use a temp file rather then
>>>        trying to buffer the response in memory?
>>>
>>>        Should I be directing this question to the developers list?
>>>
>>>        Or should I be thinking differently on solving this problem?
>>>
>>>        Please any ideas :(
>>>
>>>        Thank you
>>>        Andrew
>>>
>>>       
>
>   

Mime
View raw message