axis-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tammy Dugan <tdu...@regenstrief.org>
Subject Re: [AXIOM] How to serialize axiom document without building objects in memory
Date Fri, 29 Jun 2007 17:03:20 GMT
After doing some more testing, it looks like my solution works. Before, 
I never got an out of memory message, but the returned xml file was 
empty for a long time (20 minutes or more even though content was being 
written to it the whole time). I never let the process finish, though. I 
guess a larger amount than I expected was buffered before being written. 
I received no out of memory errors on a 46MB file so I guess everything 
is fine. I will do a little more testing to confirm this.

Tammy

Davanum Srinivas wrote:
> Ah. Sorry. need to think about that
>
> On 6/28/07, Tammy Dugan <tdugan@regenstrief.org> wrote:
>> I'm not streaming attachments. I am streaming soap body xml. Will the
>> custom dataHandler work for that too?
>>
>> Thanks,
>>
>> Tammy
>>
>> Davanum Srinivas wrote:
>> > http://marc.info/?l=axis-user&m=118282183314980&w=2
>> >
>> > On 6/28/07, Tammy Dugan <tdugan@regenstrief.org> wrote:
>> >> With a little work, I was able to get streaming soap body xml output
>> >> with input coming from a database to work for very large files 
>> (200MB).
>> >> Thanks, Ryan. However, now I want to be able to stream dynamically
>> >> created xml. Here is most of the code from my service:
>> >>
>> >>         StringDOM dom = new StringDOM();
>> >>         PipedOutputStream output = new PipedOutputStream();
>> >>         final StringDocument variableListDoc =
>> >> dom.createDocument(output);
>> >>         InputStream inputStream = new PipedInputStream(output);
>> >>         OMFactory factory = OMAbstractFactory.getOMFactory();
>> >>         OMElement resultElement =
>> >> 
>> XMLUtil.parseXMLFromStreamOM(inputStream,"getQueryVariableListResponse",true,factory);

>>
>> >>
>> >>         final StringElement resultXML = (StringElement)
>> >> variableListDoc.createElement(RESPONSE_XML_NODENAME);
>> >>         Thread thread = new Thread(new Runnable()
>> >>                 {
>> >>                         public void run()
>> >>                         {
>> >>                                 try
>> >>                                 {
>> >>                                         Connection con =
>> >> db.openConnection();
>> >>                                         StringElement result =
>> >> (StringElement) qdb.readQueryVariableList(con, db, variableListDoc);
>> >>                                         
>> resultXML.appendChild(result);
>> >>                                         result.closeBuffer();
>> >>                                 } catch (Exception e)
>> >>                                 {
>> >>
>> >> InitializationServlet.error("", e);
>> >>                                 }
>> >>                         }
>> >>                 });
>> >>
>> >>                 thread.start();
>> >>
>> >>                 return resultElement;
>> >>
>> >>
>> >> StringDom is a DOM implementation one of our developers wrote that
>> >> streams the content of a node added through an appendChild call, thus
>> >> very little is ever held in memory.
>> >> When the above service runs, it is buffering the content of
>> >> resultElement in memory and not streaming it to the response. I
>> >> copied the exact same code to a local test case and ran
>> >> serializeAndConsume on resultElement.
>> >> When I did this, everything streamed just fine. I am using a Custom
>> >> OMDataSource to create resultElement as an OMSourcedElementImpl to
>> >> facilitate the streaming. When I ran the local example that streamed,
>> >> the following method was called in my custom data source:
>> >>
>> >> public void serialize(OutputStream output, OMOutputFormat format)
>> >> throws XMLStreamException
>> >>         {
>> >>                 System.out.println("CustomDataSource
>> >> serialized(output,format)");
>> >>                 try
>> >>                 {
>> >>
>> >> org.regenstrief.util.Util.bufferedReadWrite(this.data, output, 
>> false);
>> >>                 } catch (IOException e)
>> >>                 {
>> >>                         throw new XMLStreamException(e);
>> >>                 }
>> >>         }
>> >>
>> >> where bufferedReadWrite just reads the bytes from an input stream and
>> >> writes them to an output stream in chunks. The serialization method
>> >> called when I executed the service was the following:
>> >>
>> >> public void serialize(XMLStreamWriter xmlWriter) throws
>> >> XMLStreamException
>> >>         {
>> >>                 System.out.println("CustomDataSource
>> >> serialized(xmlWriter)");
>> >>                 XMLStreamReader reader = null;
>> >>
>> >>                 try
>> >>                 {
>> >>                         StreamingOMSerializer serializer = new
>> >> StreamingOMSerializer();
>> >>                         reader =
>> >> StAXUtils.createXMLStreamReader(this.data);
>> >>                         serializer.serialize(reader, xmlWriter);//
>> >> OutOfMemory error here
>> >>                 } catch (Throwable e)
>> >>                 {
>> >>                                 InitializationServlet.error("", e);
>> >>                 }
>> >>         }
>> >>
>> >> This method would not stream the data to the output. Is there
>> >> something about StreamingOMSerializer that it is hanging while the
>> >> xml is getting dynamically built? This same method works just fine 
>> with
>> >> an InputStream from a database so why doesn't it work for a
>> >> PipedInputStream?
>> >>
>> >> Thanks,
>> >>
>> >> Tammy
>> >>
>> >>
>> >> >I've had some luck getting this to work by implementing an
>> >> OMDataSource that
>> >> >doesn't start writing XML until it is passed a writer and then using
>> >> it to
>> >> >create an OMSourcedElementImpl.  It sort of depends on the semantics
>> >> of what
>> >> >you are doing, and you have to be careful as to what methods get
>> >> called on
>> >> >the OMSourcedElementImpl (it doesn't seem to take much to cause the
>> >> whole
>> >> >thing to inflate - try to avoid doing anything that causes the
>> >> getReader()
>> >> >method to get called before you are ready to start sending stuff out
>> >> the
>> >> >door), but we've managed to get true streaming to work using
>> >> Axis2/Axiom,
>> >> >which saves a lot of transient memory.
>> >>
>> >> --
>> >> Tammy Dugan
>> >> Computer Programmer
>> >>
>> >> Regenstrief Institute, Inc.
>> >> Medical Informatics
>> >> Health Information and Translational Sciences (HITS) Building
>> >> 410 West 10th Street, Suite 2000
>> >> Indianapolis, IN 46202
>> >> Main: 317.423.5500
>> >> Fax: 317.423.5695
>> >> IU campus mail address: HS, 2000
>> >>
>> >> (317) 423 - 5541
>> >>
>> >> Confidentiality Notice: The contents of this message and any files
>> >> transmitted with it may contain confidential and/or privileged
>> >> information and are intended solely for the use of the named
>> >> addressee(s). Additionally, the information contained herein may have
>> >> been disclosed to you from medical records with confidentiality
>> >> protected by federal and state laws. Federal regulations and State
>> >> laws prohibit you from making further disclosure of such information
>> >> without the specific written consent of the person to whom the
>> >> information pertains or as otherwise permitted by such regulations. A
>> >> general authorization for the release of medical or other information
>> >> is not sufficient for this purpose.
>> >>
>> >> If you have received this message in error, please notify the sender
>> >> by return e-mail and delete the original message. Any retention,
>> >> disclosure, copying, distribution or use of this information by
>> >> anyone other than the intended recipient is strictly prohibited.
>> >>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>> >> For additional commands, e-mail: axis-user-help@ws.apache.org
>> >>
>> >>
>> >
>> >
>>
>> -- 
>> Tammy Dugan
>> Computer Programmer
>>
>> Regenstrief Institute, Inc.
>> Medical Informatics
>> Health Information and Translational Sciences (HITS) Building
>> 410 West 10th Street, Suite 2000
>> Indianapolis, IN 46202
>> Main: 317.423.5500
>> Fax: 317.423.5695
>> IU campus mail address: HS, 2000
>>
>> (317) 423 - 5541
>>
>> Confidentiality Notice: The contents of this message and any files 
>> transmitted with it may contain confidential and/or privileged 
>> information and are intended solely for the use of the named 
>> addressee(s). Additionally, the information contained herein may have 
>> been disclosed to you from medical records with confidentiality 
>> protected by federal and state laws. Federal regulations and State 
>> laws prohibit you from making further disclosure of such information 
>> without the specific written consent of the person to whom the 
>> information pertains or as otherwise permitted by such regulations. A 
>> general authorization for the release of medical or other information 
>> is not sufficient for this purpose.
>>
>> If you have received this message in error, please notify the sender 
>> by return e-mail and delete the original message. Any retention, 
>> disclosure, copying, distribution or use of this information by 
>> anyone other than the intended recipient is strictly prohibited.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-user-help@ws.apache.org
>>
>>
>
>

-- 
Tammy Dugan
Computer Programmer

Regenstrief Institute, Inc.
Medical Informatics
Health Information and Translational Sciences (HITS) Building
410 West 10th Street, Suite 2000
Indianapolis, IN 46202
Main: 317.423.5500
Fax: 317.423.5695
IU campus mail address: HS, 2000

(317) 423 - 5541

Confidentiality Notice: The contents of this message and any files transmitted with it may
contain confidential and/or privileged information and are intended solely for the use of
the named addressee(s). Additionally, the information contained herein may have been disclosed
to you from medical records with confidentiality protected by federal and state laws. Federal
regulations and State laws prohibit you from making further disclosure of such information
without the specific written consent of the person to whom the information pertains or as
otherwise permitted by such regulations. A general authorization for the release of medical
or other information is not sufficient for this purpose.
 
If you have received this message in error, please notify the sender by return e-mail and
delete the original message. Any retention, disclosure, copying, distribution or use of this
information by anyone other than the intended recipient is strictly prohibited.



---------------------------------------------------------------------
To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-user-help@ws.apache.org


Mime
View raw message