Return-Path: Delivered-To: apmail-ws-axis-user-archive@www.apache.org Received: (qmail 59265 invoked from network); 25 May 2007 05:12:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 25 May 2007 05:12:08 -0000 Received: (qmail 17200 invoked by uid 500); 25 May 2007 05:12:04 -0000 Delivered-To: apmail-ws-axis-user-archive@ws.apache.org Received: (qmail 16848 invoked by uid 500); 25 May 2007 05:12:03 -0000 Mailing-List: contact axis-user-help@ws.apache.org; run by ezmlm Precedence: bulk Reply-To: axis-user@ws.apache.org list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list axis-user@ws.apache.org Received: (qmail 16837 invoked by uid 99); 25 May 2007 05:12:03 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 May 2007 22:12:03 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [206.123.75.163] (HELO sosnoski.com) (206.123.75.163) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 May 2007 22:11:56 -0700 Received: from [192.168.0.2] (ip-58-28-158-218.ubs-dsl.xnet.co.nz [58.28.158.218]) (authenticated bits=0) by sosnoski.com (8.12.10/8.12.10) with ESMTP id l4P5BVlm015728 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 25 May 2007 01:11:34 -0400 Message-ID: <46567002.4040104@sosnoski.com> Date: Fri, 25 May 2007 17:11:30 +1200 From: Dennis Sosnoski Organization: Sosnoski Software User-Agent: Thunderbird 1.5.0.4 (X11/20060516) MIME-Version: 1.0 To: axis-user@ws.apache.org Subject: Re: MTOM error "java.lang.OutOfMemoryError: Java heap space" References: <92835690@web.de> <19e0530f0705230620t7b7e08e2r56afc5e0e76d2b69@mail.gmail.com> <46544C5E.8070808@regenstrief.org> <19e0530f0705230719k19754d97x751fcd35c554e878@mail.gmail.com> <46545256.1020503@regenstrief.org> <4654A537.1050907@sosnoski.com> <4655DCA9.70404@regenstrief.org> <4655F091.3060007@sosnoski.com> <88f5d710705241315y31a5cef9va04f216fab59a2ed@mail.gmail.com> In-Reply-To: <88f5d710705241315y31a5cef9va04f216fab59a2ed@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Hi Paul, I'm not sure exactly what you mean with this code. This thread has been repurposed a time or two, but the question from Tammy was how to stream out data from a database as the body of a response. So the data is not in the form of an OMElement, and the point of using OMDataSource is to avoid building it as an OMElement. The data has to be supplied as the AXIOM representation is being written to the output stream, which is what OMDataSource allows - it's basically a placeholder that says XML output will be generated on demand. If you *do* build the data as a full AXIOM model you'll run out of memory pretty quickly. AXIOM is relatively fat by comparison with most other document models, and from what I've seen takes at least 10 times the size of the document in memory usage. So the only practical way to handle large documents (say >10 MB) is to stream the data using OMDataSource or an equivalent. Incidentally, note that this also means that anyone using WS-Security is going to effectively be limited to document sizes of about 10 MB or so. Again based on what I've seen, Rampart always ends up building the full AXIOM model even if the only thing you're doing is using a TimeStamp. I haven't verified this, but I suspect roughly the same size limit will apply for attachment handling - expanding the tree will inline the attachment data as base64 text, so the fact that it's actually transmitted as a binary blob doesn't help when Rampart is engaged. - Dennis -- Dennis M. Sosnoski SOA and Web Services in Java Axis2 Training and Consulting http://www.sosnoski.com - http://www.sosnoski.co.nz Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117 Paul Fremantle wrote: > Dennis > > What about getting to the actual StAX? That would seem efficient to > me. Or does it go wrong when you write out the new OMElement? > > public OMElement sayHello(OMElement element) > { > XMLStreamReader xsr = element.getXMLStreamReaderWithoutCaching(); > // use xsr > > > // create a new StAX stream reader > > XMLStreamReader xsrOut = ... > > StAXOMBuilder builder = new StAXOMBuilder(xsrOut); > return builder.getDocumentElement(); > > } > > > On 5/24/07, *Dennis Sosnoski* > wrote: > > The easiest way to write data to the output without first creating a > copy in memory is to use OMDataSource, as dims had said. I was adding > some details on how OMDataSource is used, and pointing you at one > example of its use in the Axis2 code. > > OMDataSource is not really designed for end users, so you're going to > need to dig into the AXIOM and/or Axis2 code to understand how to > use this. > > - Dennis > > -- > Dennis M. Sosnoski > SOA and Web Services in Java > Axis2 Training and Consulting > http://www.sosnoski.com - http://www.sosnoski.co.nz > Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117 > > > > Tammy Dugan wrote: > > I still don't understand what to do. I don't use WSDLs for my Axis2 > > services. I just used the following form: > > > > public OMElement sayHello(OMElement element) { > > return method; > > } > > > > Something inside Axis2 after the return is buffering the whole > output > > string into memory. Are you telling me I can't use this method of > > service building and must use the WSDL way? > > > > > > Tammy > > > > Dennis Sosnoski wrote: > >> Hi Tammy, > >> > >> OMDataSource is the technique used by data binding code to tie into > >> the Axis2 output serialization. This allows the data binding to > >> effectively write directly to the output, as long as the DOM model > >> doesn't need to be constructed for some reason (such as > Rampart). The > >> JiBX implementation is at > >> modules/jibx/src/org/apache/axis2/jibx/JiBXDataSource.java in the > >> Axis2 source code. The code that actually uses this is generated > >> through WSDL2Java, and if you look at a generated JiBX stub or > >> message receiver you will see the JiBXDataSource being used for > >> objects being passed as part of the message. > >> > >> - Dennis > >> > >> Dennis M. Sosnoski > >> SOA and Web Services in Java > >> Axis2 Training and Consulting > >> http://www.sosnoski.com - http://www.sosnoski.co.nz > >> Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117 > >> > >> > >> > >> Tammy Dugan wrote: > >>> Can you tell me a specific test case or supply a snippet of > code so > >>> I know how to use an OMDataSource with an inputStream? I don't > know > >>> what classes implement this interface. > >>> > >>> Tammy > >>> > >>> Davanum Srinivas wrote: > >>>> Hmm...Please *don't* do this :) You are better off working with > >>>> OMDataSource since it will avoid the overhead of actually > creating the > >>>> DOM document in memory if it is not needed. Please check the > axiom > >>>> test cases for info on how to work with OMDataSource. > >>>> > >>>> thanks, > >>>> dims > >>>> > >>>> On 5/23/07, Tammy Dugan > wrote: > >>>>> I have been having a similar problem but with the actual xml in > >>>>> the body > >>>>> of the SOAP message. I am trying to stream xml from a > database to the > >>>>> service output but am getting out of memory errors because > >>>>> something in > >>>>> Axis2 is buffering the data. The soap xml I am sending back > is very > >>>>> large (>100 MB). Here is some code that illustrates what I am > >>>>> trying to do: > >>>>> > >>>>> package org.regenstrief.query_tests; > >>>>> > >>>>> import java.io.InputStream; > >>>>> > >>>>> import javax.xml.stream.XMLInputFactory; > >>>>> import javax.xml.stream.XMLStreamReader; > >>>>> > >>>>> import org.apache.axiom.om.OMDocument; > >>>>> import org.apache.axiom.om.OMElement; > >>>>> import org.apache.axiom.om.impl.builder.StAXOMBuilder; > >>>>> import org.apache.axiom.om.impl.dom.DocumentImpl; > >>>>> import org.apache.axiom.om.impl.dom.factory.OMDOMFactory ; > >>>>> import org.w3c.dom.Element; > >>>>> > >>>>> public class TestCase > >>>>> { > >>>>> //service method that would get called by axis2 > >>>>> public OMElement processBody() throws Exception > >>>>> { > >>>>> DocumentImpl responseDoc = new DocumentImpl(new > >>>>> OMDOMFactory()); > >>>>> Element resultXML = > >>>>> responseDoc.createElement("queryDatasetReturnResponse"); > >>>>> > >>>>> OMElement rawdataNode = readDataSetXMLToDom(); > >>>>> ((OMElement) resultXML).addChild(rawdataNode); > >>>>> > >>>>> return (OMElement) resultXML; > >>>>> } > >>>>> > >>>>> private static OMElement readDataSetXMLToDom() throws > Exception > >>>>> { > >>>>> OMElement resultElement = null; > >>>>> OMDocument datasetDoc = null; > >>>>> > >>>>> InputStream inputStream = null; //input stream > pulled from > >>>>> database > >>>>> > >>>>> if (inputStream != null) > >>>>> { > >>>>> XMLStreamReader parser = > >>>>> > XMLInputFactory.newInstance().createXMLStreamReader(inputStream); > >>>>> StAXOMBuilder builder = new StAXOMBuilder(parser); > >>>>> datasetDoc = builder.getDocument(); > >>>>> resultElement = datasetDoc.getOMDocumentElement(); > >>>>> > >>>>> } > >>>>> > >>>>> return resultElement; > >>>>> } > >>>>> > >>>>> } > >>>>> > >>>>> > >>>>> Does axis2 support unlimited xml in the soap body? > >>>>> > >>>>> > >>>>> Tammy > >>>>> > >>>>> Davanum Srinivas wrote: > >>>>> > Jochen, > >>>>> > > >>>>> > we are not taking about the SAAJ API, check the Attachments > >>>>> class in > >>>>> > axiom. There is a method called > getIncomingAttachmentStreams(). > >>>>> Also > >>>>> > check the axiom test harness it shows how to use that API. > this > >>>>> will > >>>>> > give you direct access to the streams and you can do whatever > >>>>> you want > >>>>> > by yourself. Note that this API has not been thoroughly > >>>>> exercised, so > >>>>> > you may need to experiment/fix-stuff and help us make it > better. > >>>>> > > >>>>> > thanks, > >>>>> > dims > >>>>> > > >>>>> > On 5/23/07, Jochen Zink > wrote: > >>>>> >> Sorry, I did not want to affront you. A big sorry for that! > >>>>> >> > >>>>> >> I know it is difficult to handle large attachments without > >>>>> caching it > >>>>> >> in files. > >>>>> >> > >>>>> >> With caching attachments in files, there can(!) be some > >>>>> problems with > >>>>> >> security configurations on some applicationservers. It > can be > >>>>> >> forbidden for webapplications to write any files to > harddisc. This > >>>>> >> could run in problems, if some people want to use axis2 > in high > >>>>> >> security environments, this could be a problem. A direct > >>>>> streaming of > >>>>> >> the attachments to the Service Implementation would be > solve the > >>>>> >> problem... After taking a look to SAAJ I'm nearly sure, > that a > >>>>> direct > >>>>> >> streaming is imposible with this API. > >>>>> >> > >>>>> >> Regards! > >>>>> >> Jochen > >>>>> >> > >>>>> >> > >>>>> >> > -----Urspr�ngliche Nachricht----- > >>>>> >> > Von: axis-user@ws.apache.org > > >>>>> >> > Gesendet: 23.05.07 10:15:30 > >>>>> >> > An: axis-user@ws.apache.org > > >>>>> >> > Betreff: Re: MTOM error " java.lang.OutOfMemoryError: Java > >>>>> heap space" > >>>>> >> > >>>>> >> > >>>>> >> > > >>>>> >> > On 5/23/07, Jochen Zink < jochenlist@web.de > > wrote: > >>>>> >> > > > >>>>> >> > > This is the only way to handle large attachments int > axis2, > >>>>> *sadly*. > >>>>> >> > Feel free to contribute or implement if you have ideas for > >>>>> better > >>>>> >> > mechanisms for handling large attachments... There was an > >>>>> earlier > >>>>> >> > effort to port the attachment streaming mechanisms from > >>>>> Axis1.. But it > >>>>> >> > was never completed.. > >>>>> >> > > >>>>> >> > Thanks, > >>>>> >> > Thilina > >>>>> >> > > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > -----Urspr�ngliche Nachricht----- > >>>>> >> > > > Von: axis-user@ws.apache.org > > >>>>> >> > > > Gesendet: 23.05.07 07:23:12 > >>>>> >> > > > An: "axis user" > > >>>>> >> > > > Betreff: MTOM error " java.lang.OutOfMemoryError: Java > >>>>> heap space" > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > Hi friends, > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > I am running sample application "mtom" from > standard > >>>>> axis2 > >>>>> >> samples [axis2-1.1.1\samples\mtom]. This sample is > working for > >>>>> >> sending attachment of size 2-3 MB, but giving error for more > >>>>> than 3MB > >>>>> >> size of files. > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > I want to send file of size 50-100 MB. > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > error is as follows > >>>>> >> > > > > >>>>> >> > > > [java] Exception in thread "main" > >>>>> >> org.apache.axis2.AxisFault: java.lang.Out > >>>>> >> > > > OfMemoryError: Java heap space > >>>>> >> > > > [java] at > >>>>> >> org.apache.axis2.description.OutInAxisOperationClient.send(Ou > >>>>> >> > > > tInAxisOperation.java:271) > >>>>> >> > > > [java] at > >>>>> >> org.apache.axis2.description.OutInAxisOperationClient.execute > >>>>> >> > > > (OutInAxisOperation.java:202) > >>>>> >> > > > [java] at > >>>>> >> sample.mtom.service.MTOMSampleStub.attachment(Unknown Source) > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > [java] at > >>>>> >> sample.mtom.client.Client.transferFile(Unknown Source) > >>>>> >> > > > [java] at > sample.mtom.client.Client.main(Unknown > >>>>> Source) > >>>>> >> > > > [java] Java Result: 1 > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > I have enabled "MTOM" & "filecaching" both in the > client > >>>>> side, > >>>>> >> but not server side caching means in " axis2.xml". > >>>>> >> > > > > >>>>> >> > > > 1) Can some one figureout solution? > >>>>> >> > > > > >>>>> >> > > > 2) Can we send file as attachment of size 50MB - 1GB? > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > > >>>>> >> > > > With best regards > >>>>> >> > > > > >>>>> >> > > > From > >>>>> >> > > > Vikas R. Khengare > >>>>> >> > > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> _______________________________________________________________ > >>>>> >> > > SMS schreiben mit WEB.DE FreeMail - > einfach, schnell und > >>>>> >> > > kostenguenstig. Jetzt gleich testen! > >>>>> http://f.web.de/?mc=021192 > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> > >>>>> > --------------------------------------------------------------------- > >>>>> >> > > To unsubscribe, e-mail: > axis-user-unsubscribe@ws.apache.org > > >>>>> >> > > For additional commands, e-mail: > axis-user-help@ws.apache.org > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > >>>>> >> > > >>>>> >> > -- > >>>>> >> > Thilina Gunarathne - http://www.wso2.com - > >>>>> >> http://thilinag.blogspot.com > >>>>> >> > > >>>>> >> > > >>>>> > --------------------------------------------------------------------- > >>>>> >> > To unsubscribe, e-mail: > axis-user-unsubscribe@ws.apache.org > > >>>>> >> > For additional commands, e-mail: > axis-user-help@ws.apache.org > >>>>> >> > > >>>>> >> > > >>>>> >> > >>>>> >> > >>>>> >> > _______________________________________________________________ > >>>>> >> SMS schreiben mit WEB.DE FreeMail - > einfach, schnell und > >>>>> >> kostenguenstig. Jetzt gleich testen! > http://f.web.de/?mc=021192 > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> > --------------------------------------------------------------------- > >>>>> >> To unsubscribe, e-mail: > axis-user-unsubscribe@ws.apache.org > > >>>>> >> For additional commands, e-mail: > axis-user-help@ws.apache.org > >>>>> >> > >>>>> >> > >>>>> > > >>>>> > > >>>>> > >>>>> -- > >>>>> Tammy Dugan > >>>>> Computer Programmer > >>>>> > >>>>> Regenstrief Institute, Inc. > >>>>> Medical Informatics > >>>>> Health Information and Translational Sciences (HITS) Building > >>>>> 410 West 10th Street, Suite 2000 > >>>>> Indianapolis, IN 46202 > >>>>> Main: 317.423.5500 > >>>>> Fax: 317.423.5695 > >>>>> IU campus mail address: HS, 2000 > >>>>> > >>>>> (317) 423 - 5541 > >>>>> > >>>>> Confidentiality Notice: The contents of this message and any > files > >>>>> transmitted with it may contain confidential and/or privileged > >>>>> information and are intended solely for the use of the named > >>>>> addressee(s). Additionally, the information contained herein may > >>>>> have been disclosed to you from medical records with > >>>>> confidentiality protected by federal and state laws. Federal > >>>>> regulations and State laws prohibit you from making further > >>>>> disclosure of such information without the specific written > >>>>> consent of the person to whom the information pertains or as > >>>>> otherwise permitted by such regulations. A general authorization > >>>>> for the release of medical or other information is not > sufficient > >>>>> for this purpose. > >>>>> > >>>>> If you have received this message in error, please notify the > >>>>> sender by return e-mail and delete the original message. Any > >>>>> retention, disclosure, copying, distribution or use of this > >>>>> information by anyone other than the intended recipient is > >>>>> strictly prohibited. > >>>>> > >>>>> > >>>>> > >>>>> > --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org > > >>>>> For additional commands, e-mail: > axis-user-help@ws.apache.org > >>>>> > >>>>> > >>>> > >>>> > >> > >> > --------------------------------------------------------------------- > >> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org > > >> For additional commands, e-mail: axis-user-help@ws.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org > > For additional commands, e-mail: axis-user-help@ws.apache.org > > > > > > -- > Paul Fremantle > Co-Founder and VP of Technical Sales, WSO2 > OASIS WS-RX TC Co-chair > > blog: http://pzf.fremantle.org > paul@wso2.com > > "Oxygenating the Web Service Platform", www.wso2.com > --------------------------------------------------------------------- To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org For additional commands, e-mail: axis-user-help@ws.apache.org