axis-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Sosnoski <...@sosnoski.com>
Subject Re: MTOM error "java.lang.OutOfMemoryError: Java heap space"
Date Fri, 25 May 2007 05:11:30 GMT
Hi Paul,

I'm not sure exactly what you mean with this code. This thread has been 
repurposed a time or two, but the question from Tammy was how to stream 
out data from a database as the body of a response. So the data is not 
in the form of an OMElement, and the point of using OMDataSource is to 
avoid building it as an OMElement. The data has to be supplied as the 
AXIOM representation is being written to the output stream, which is 
what OMDataSource allows - it's basically a placeholder that says XML 
output will be generated on demand.

If you *do* build the data as a full AXIOM model you'll run out of 
memory pretty quickly. AXIOM is relatively fat by comparison with most 
other document models, and from what I've seen takes at least 10 times 
the size of the document in memory usage. So the only practical way to 
handle large documents (say >10 MB) is to stream the data using 
OMDataSource or an equivalent.

Incidentally, note that this also means that anyone using WS-Security is 
going to effectively be limited to document sizes of about 10 MB or so. 
Again based on what I've seen, Rampart always ends up building the full 
AXIOM model even if the only thing you're doing is using a TimeStamp. I 
haven't verified this, but I suspect roughly the same size limit will 
apply for attachment handling - expanding the tree will inline the 
attachment data as base64 text, so the fact that it's actually 
transmitted as a binary blob doesn't help when Rampart is engaged.

  - Dennis

-- 
Dennis M. Sosnoski
SOA and Web Services in Java
Axis2 Training and Consulting
http://www.sosnoski.com - http://www.sosnoski.co.nz
Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117



Paul Fremantle wrote:
> Dennis
>
> What about getting to the actual StAX? That would seem efficient to 
> me. Or does it go wrong when you write out the new OMElement?
>
> public OMElement sayHello(OMElement element)
> {
>    XMLStreamReader xsr =  element.getXMLStreamReaderWithoutCaching();
>   // use xsr
>
>
>   //   create a new StAX stream reader
>
>   XMLStreamReader xsrOut = ...
>
>   StAXOMBuilder builder = new StAXOMBuilder(xsrOut);
>    return builder.getDocumentElement();
>
> }
>
>
> On 5/24/07, *Dennis Sosnoski* <dms@sosnoski.com 
> <mailto:dms@sosnoski.com>> wrote:
>
>     The easiest way to write data to the output without first creating a
>     copy in memory is to use OMDataSource, as dims had said. I was adding
>     some details on how OMDataSource is used, and pointing you at one
>     example of its use in the Axis2 code.
>
>     OMDataSource is not really designed for end users, so you're going to
>     need to dig into the AXIOM and/or Axis2 code to understand how to
>     use this.
>
>       - Dennis
>
>     --
>     Dennis M. Sosnoski
>     SOA and Web Services in Java
>     Axis2 Training and Consulting
>     http://www.sosnoski.com - http://www.sosnoski.co.nz
>     Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
>
>
>
>     Tammy Dugan wrote:
>     > I still don't understand what to do. I don't use WSDLs for my Axis2
>     > services. I just used the following form:
>     >
>     > public OMElement sayHello(OMElement element) {
>     >            return method;
>     > }
>     >
>     > Something inside Axis2 after the return is buffering the whole
>     output
>     > string into memory. Are you telling me I can't use this method of
>     > service building and must use the WSDL way?
>     >
>     >
>     > Tammy
>     >
>     > Dennis Sosnoski wrote:
>     >> Hi Tammy,
>     >>
>     >> OMDataSource is the technique used by data binding code to tie into
>     >> the Axis2 output serialization. This allows the data binding to
>     >> effectively write directly to the output, as long as the DOM model
>     >> doesn't need to be constructed for some reason (such as
>     Rampart). The
>     >> JiBX implementation is at
>     >> modules/jibx/src/org/apache/axis2/jibx/JiBXDataSource.java in the
>     >> Axis2 source code. The code that actually uses this is generated
>     >> through WSDL2Java, and if you look at a generated JiBX stub or
>     >> message receiver you will see the JiBXDataSource being used for
>     >> objects being passed as part of the message.
>     >>
>     >>  - Dennis
>     >>
>     >> Dennis M. Sosnoski
>     >> SOA and Web Services in Java
>     >> Axis2 Training and Consulting
>     >> http://www.sosnoski.com - http://www.sosnoski.co.nz
>     >> Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117
>     >>
>     >>
>     >>
>     >> Tammy Dugan wrote:
>     >>> Can you tell me a specific test case or supply a snippet of
>     code so
>     >>> I know how to use an OMDataSource with an inputStream? I don't
>     know
>     >>> what classes implement this interface.
>     >>>
>     >>> Tammy
>     >>>
>     >>> Davanum Srinivas wrote:
>     >>>> Hmm...Please *don't* do this :) You are better off working with
>     >>>> OMDataSource since it will avoid the overhead of actually
>     creating the
>     >>>> DOM document in memory if it is not needed. Please check the
>     axiom
>     >>>> test cases for info on how to work with OMDataSource.
>     >>>>
>     >>>> thanks,
>     >>>> dims
>     >>>>
>     >>>> On 5/23/07, Tammy Dugan <tdugan@regenstrief.org
>     <mailto:tdugan@regenstrief.org>> wrote:
>     >>>>> I have been having a similar problem but with the actual xml
in
>     >>>>> the body
>     >>>>> of the SOAP message. I am trying to stream xml from a
>     database to the
>     >>>>> service output but am getting out of memory errors because
>     >>>>> something in
>     >>>>> Axis2 is buffering the data. The soap xml I am sending back
>     is very
>     >>>>> large (>100 MB). Here is some code that illustrates what
I am
>     >>>>> trying to do:
>     >>>>>
>     >>>>> package org.regenstrief.query_tests;
>     >>>>>
>     >>>>> import java.io.InputStream;
>     >>>>>
>     >>>>> import javax.xml.stream.XMLInputFactory;
>     >>>>> import javax.xml.stream.XMLStreamReader;
>     >>>>>
>     >>>>> import org.apache.axiom.om.OMDocument;
>     >>>>> import org.apache.axiom.om.OMElement;
>     >>>>> import org.apache.axiom.om.impl.builder.StAXOMBuilder;
>     >>>>> import org.apache.axiom.om.impl.dom.DocumentImpl;
>     >>>>> import org.apache.axiom.om.impl.dom.factory.OMDOMFactory ;
>     >>>>> import org.w3c.dom.Element;
>     >>>>>
>     >>>>> public class TestCase
>     >>>>> {
>     >>>>>     //service method that would get called by axis2
>     >>>>>     public OMElement processBody() throws Exception
>     >>>>>     {
>     >>>>>         DocumentImpl responseDoc = new DocumentImpl(new
>     >>>>> OMDOMFactory());
>     >>>>>         Element resultXML =
>     >>>>> responseDoc.createElement("queryDatasetReturnResponse");
>     >>>>>
>     >>>>>         OMElement rawdataNode = readDataSetXMLToDom();
>     >>>>>         ((OMElement) resultXML).addChild(rawdataNode);
>     >>>>>
>     >>>>>         return (OMElement) resultXML;
>     >>>>>     }
>     >>>>>
>     >>>>>     private static OMElement readDataSetXMLToDom() throws
>     Exception
>     >>>>>     {
>     >>>>>         OMElement resultElement = null;
>     >>>>>         OMDocument datasetDoc = null;
>     >>>>>
>     >>>>>         InputStream inputStream = null; //input stream
>     pulled from
>     >>>>> database
>     >>>>>
>     >>>>>         if (inputStream != null)
>     >>>>>         {
>     >>>>>             XMLStreamReader parser =
>     >>>>>
>     XMLInputFactory.newInstance().createXMLStreamReader(inputStream);
>     >>>>>             StAXOMBuilder builder = new StAXOMBuilder(parser);
>     >>>>>             datasetDoc = builder.getDocument();
>     >>>>>             resultElement = datasetDoc.getOMDocumentElement();
>     >>>>>
>     >>>>>         }
>     >>>>>
>     >>>>>         return resultElement;
>     >>>>>     }
>     >>>>>
>     >>>>> }
>     >>>>>
>     >>>>>
>     >>>>> Does axis2 support unlimited xml in the soap body?
>     >>>>>
>     >>>>>
>     >>>>> Tammy
>     >>>>>
>     >>>>> Davanum Srinivas wrote:
>     >>>>> > Jochen,
>     >>>>> >
>     >>>>> > we are not taking about the SAAJ API, check the Attachments
>     >>>>> class in
>     >>>>> > axiom. There is a method called
>     getIncomingAttachmentStreams().
>     >>>>> Also
>     >>>>> > check the axiom test harness it shows how to use that API.
>     this
>     >>>>> will
>     >>>>> > give you direct access to the streams and you can do whatever
>     >>>>> you want
>     >>>>> > by yourself. Note that this API has not been thoroughly
>     >>>>> exercised, so
>     >>>>> > you may need to experiment/fix-stuff and help us make it
>     better.
>     >>>>> >
>     >>>>> > thanks,
>     >>>>> > dims
>     >>>>> >
>     >>>>> > On 5/23/07, Jochen Zink <jochenlist@web.de
>     <mailto:jochenlist@web.de>> wrote:
>     >>>>> >> Sorry, I did not want to affront you. A big sorry for
that!
>     >>>>> >>
>     >>>>> >> I know it is difficult to handle large attachments
without
>     >>>>> caching it
>     >>>>> >> in files.
>     >>>>> >>
>     >>>>> >> With caching attachments in files, there can(!) be
some
>     >>>>> problems with
>     >>>>> >> security configurations on some applicationservers.
It
>     can be
>     >>>>> >> forbidden for webapplications to write any files to
>     harddisc. This
>     >>>>> >> could run in problems, if some people want to use axis2
>     in high
>     >>>>> >> security environments, this could be a problem. A direct
>     >>>>> streaming of
>     >>>>> >> the attachments to the Service Implementation would
be
>     solve the
>     >>>>> >> problem... After taking a look to SAAJ I'm nearly sure,
>     that a
>     >>>>> direct
>     >>>>> >> streaming is imposible with this API.
>     >>>>> >>
>     >>>>> >> Regards!
>     >>>>> >> Jochen
>     >>>>> >>
>     >>>>> >>
>     >>>>> >> > -----Urspr√ľngliche Nachricht-----
>     >>>>> >> > Von: axis-user@ws.apache.org
>     <mailto:axis-user@ws.apache.org>
>     >>>>> >> > Gesendet: 23.05.07 10:15:30
>     >>>>> >> > An: axis-user@ws.apache.org
>     <mailto:axis-user@ws.apache.org>
>     >>>>> >> > Betreff: Re: MTOM error " java.lang.OutOfMemoryError:
Java
>     >>>>> heap space"
>     >>>>> >>
>     >>>>> >>
>     >>>>> >> >
>     >>>>> >> > On 5/23/07, Jochen Zink < jochenlist@web.de
>     <mailto:jochenlist@web.de>> wrote:
>     >>>>> >> > >
>     >>>>> >> > > This is the only way to handle large attachments
int
>     axis2,
>     >>>>> *sadly*.
>     >>>>> >> > Feel free to contribute or implement if you have
ideas for
>     >>>>> better
>     >>>>> >> > mechanisms for handling large attachments... There
was an
>     >>>>> earlier
>     >>>>> >> > effort to port the attachment streaming mechanisms
from
>     >>>>> Axis1.. But it
>     >>>>> >> > was never completed..
>     >>>>> >> >
>     >>>>> >> > Thanks,
>     >>>>> >> > Thilina
>     >>>>> >> >
>     >>>>> >> > >
>     >>>>> >> > >
>     >>>>> >> > > > -----Urspr√ľngliche Nachricht-----
>     >>>>> >> > > > Von: axis-user@ws.apache.org
>     <mailto:axis-user@ws.apache.org>
>     >>>>> >> > > > Gesendet: 23.05.07 07:23:12
>     >>>>> >> > > > An: "axis user" <axis-user@ws.apache.org
>     <mailto:axis-user@ws.apache.org>>
>     >>>>> >> > > > Betreff: MTOM error " java.lang.OutOfMemoryError:
Java
>     >>>>> heap space"
>     >>>>> >> > >
>     >>>>> >> > >
>     >>>>> >> > > > Hi friends,
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >     I am running sample application
"mtom" from
>     standard
>     >>>>> axis2
>     >>>>> >> samples [axis2-1.1.1\samples\mtom]. This sample is
>     working for
>     >>>>> >> sending attachment of size 2-3 MB, but giving error
for more
>     >>>>> than 3MB
>     >>>>> >> size of files.
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > > I want to send file of size 50-100 MB.
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > > error is as follows
>     >>>>> >> > > >
>     >>>>> >> > > >      [java] Exception in thread "main"
>     >>>>> >> org.apache.axis2.AxisFault: java.lang.Out
>     >>>>> >> > > > OfMemoryError: Java heap space
>     >>>>> >> > > >      [java]     at
>     >>>>> >> org.apache.axis2.description.OutInAxisOperationClient.send(Ou
>     >>>>> >> > > > tInAxisOperation.java:271)
>     >>>>> >> > > >      [java]     at
>     >>>>> >> org.apache.axis2.description.OutInAxisOperationClient.execute
>     >>>>> >> > > > (OutInAxisOperation.java:202)
>     >>>>> >> > > >      [java]     at
>     >>>>> >> sample.mtom.service.MTOMSampleStub.attachment(Unknown
Source)
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >      [java]     at
>     >>>>> >> sample.mtom.client.Client.transferFile(Unknown Source)
>     >>>>> >> > > >      [java]     at
>     sample.mtom.client.Client.main(Unknown
>     >>>>> Source)
>     >>>>> >> > > >      [java] Java Result: 1
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > > I have enabled "MTOM" & "filecaching"
both in the
>     client
>     >>>>> side,
>     >>>>> >> but not server side caching means in " axis2.xml".
>     >>>>> >> > > >
>     >>>>> >> > > > 1) Can some one figureout solution?
>     >>>>> >> > > >
>     >>>>> >> > > > 2) Can we send file as attachment of
size 50MB - 1GB?
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > >
>     >>>>> >> > > > With best regards
>     >>>>> >> > > >
>     >>>>> >> > > > From
>     >>>>> >> > > > Vikas R. Khengare
>     >>>>> >> > > >
>     >>>>> >> > >
>     >>>>> >> > >
>     >>>>> >> > >
>     >>>>> _______________________________________________________________
>     >>>>> >> > > SMS schreiben mit WEB.DE <http://WEB.DE>
FreeMail -
>     einfach, schnell und
>     >>>>> >> > > kostenguenstig. Jetzt gleich testen!
>     >>>>> http://f.web.de/?mc=021192
>     >>>>> >> > >
>     >>>>> >> > >
>     >>>>> >> > >
>     >>>>> >>
>     >>>>>
>     ---------------------------------------------------------------------
>     >>>>> >> > > To unsubscribe, e-mail:
>     axis-user-unsubscribe@ws.apache.org
>     <mailto:axis-user-unsubscribe@ws.apache.org>
>     >>>>> >> > > For additional commands, e-mail:
>     axis-user-help@ws.apache.org <mailto:axis-user-help@ws.apache.org>
>     >>>>> >> > >
>     >>>>> >> > >
>     >>>>> >> >
>     >>>>> >> >
>     >>>>> >> > --
>     >>>>> >> > Thilina Gunarathne  -   http://www.wso2.com -
>     >>>>> >> http://thilinag.blogspot.com
>     >>>>> >> >
>     >>>>> >> >
>     >>>>>
>     ---------------------------------------------------------------------
>     >>>>> >> > To unsubscribe, e-mail:
>     axis-user-unsubscribe@ws.apache.org
>     <mailto:axis-user-unsubscribe@ws.apache.org>
>     >>>>> >> > For additional commands, e-mail:
>     axis-user-help@ws.apache.org <mailto:axis-user-help@ws.apache.org>
>     >>>>> >> >
>     >>>>> >> >
>     >>>>> >>
>     >>>>> >>
>     >>>>> >>
>     _______________________________________________________________
>     >>>>> >> SMS schreiben mit WEB.DE <http://WEB.DE> FreeMail
-
>     einfach, schnell und
>     >>>>> >> kostenguenstig. Jetzt gleich testen!
>     http://f.web.de/?mc=021192
>     >>>>> >>
>     >>>>> >>
>     >>>>> >>
>     >>>>>
>     ---------------------------------------------------------------------
>     >>>>> >> To unsubscribe, e-mail:
>     axis-user-unsubscribe@ws.apache.org
>     <mailto:axis-user-unsubscribe@ws.apache.org>
>     >>>>> >> For additional commands, e-mail:
>     axis-user-help@ws.apache.org <mailto:axis-user-help@ws.apache.org>
>     >>>>> >>
>     >>>>> >>
>     >>>>> >
>     >>>>> >
>     >>>>>
>     >>>>> --
>     >>>>> Tammy Dugan
>     >>>>> Computer Programmer
>     >>>>>
>     >>>>> Regenstrief Institute, Inc.
>     >>>>> Medical Informatics
>     >>>>> Health Information and Translational Sciences (HITS) Building
>     >>>>> 410 West 10th Street, Suite 2000
>     >>>>> Indianapolis, IN 46202
>     >>>>> Main: 317.423.5500
>     >>>>> Fax: 317.423.5695
>     >>>>> IU campus mail address: HS, 2000
>     >>>>>
>     >>>>> (317) 423 - 5541
>     >>>>>
>     >>>>> Confidentiality Notice: The contents of this message and any
>     files
>     >>>>> transmitted with it may contain confidential and/or privileged
>     >>>>> information and are intended solely for the use of the named
>     >>>>> addressee(s). Additionally, the information contained herein
may
>     >>>>> have been disclosed to you from medical records with
>     >>>>> confidentiality protected by federal and state laws. Federal
>     >>>>> regulations and State laws prohibit you from making further
>     >>>>> disclosure of such information without the specific written
>     >>>>> consent of the person to whom the information pertains or as
>     >>>>> otherwise permitted by such regulations. A general authorization
>     >>>>> for the release of medical or other information is not
>     sufficient
>     >>>>> for this purpose.
>     >>>>>
>     >>>>> If you have received this message in error, please notify the
>     >>>>> sender by return e-mail and delete the original message. Any
>     >>>>> retention, disclosure, copying, distribution or use of this
>     >>>>> information by anyone other than the intended recipient is
>     >>>>> strictly prohibited.
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>>
>     ---------------------------------------------------------------------
>     >>>>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>     <mailto:axis-user-unsubscribe@ws.apache.org>
>     >>>>> For additional commands, e-mail:
>     axis-user-help@ws.apache.org <mailto:axis-user-help@ws.apache.org>
>     >>>>>
>     >>>>>
>     >>>>
>     >>>>
>     >>
>     >>
>     ---------------------------------------------------------------------
>     >> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>     <mailto:axis-user-unsubscribe@ws.apache.org>
>     >> For additional commands, e-mail: axis-user-help@ws.apache.org
>     <mailto:axis-user-help@ws.apache.org>
>     >
>
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>     <mailto:axis-user-unsubscribe@ws.apache.org>
>     For additional commands, e-mail: axis-user-help@ws.apache.org
>     <mailto:axis-user-help@ws.apache.org>
>
>
>
>
> -- 
> Paul Fremantle
> Co-Founder and VP of Technical Sales, WSO2
> OASIS WS-RX TC Co-chair
>
> blog: http://pzf.fremantle.org
> paul@wso2.com <mailto:paul@wso2.com>
>
> "Oxygenating the Web Service Platform", www.wso2.com 
> <http://www.wso2.com> 

---------------------------------------------------------------------
To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-user-help@ws.apache.org


Mime
View raw message