axis-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tammy Dugan <tdu...@regenstrief.org>
Subject Re: [AXIOM] How to serialize axiom document without building objects in memory
Date Fri, 29 Jun 2007 17:58:11 GMT
Disregard my comment on the large buffer size. I found an error with my 
test sending program that was buffering the response. Anyway, the piped 
solution seems to work just fine and I tested the piping for a file up 
to 333MB in size. Thus, I can successfully stream data in an OMElement 
from the database or from dynamically generated xml. Thanks, Davanum, 
and everyone else for your help and being patient with me while I 
learning how Axis2 works. :)

In summary, these were the key steps:

1. create an OMSourcedElementImpl and be VERY CAREFUL not to manipulate 
it too much or the whole thing will build into memory. You can append it 
to a parent without it building, though, which is useful. Here is how I 
made it:

OMNamespace ns = new OMNamespaceImpl("","");
String localName = "testing"; //name of root element from xml input stream
OMFactory factory = OMAbstractFactory.getOMFactory();
OMDataSource dataSource = new CustomDataSource(xmlDefinition);
return new OMSourcedElementImpl(localName, ns, factory, dataSource);

2. create a custom data source that streams the output. Here is the 
datasource I created:

public class CustomDataSource implements OMDataSource
{
    private final InputStream data;

    public CustomDataSource(InputStream data)
    { this.data = data; }

    public void serialize(XMLStreamWriter xmlWriter) throws 
XMLStreamException
    {
        XMLStreamReader reader = null;

        try
        {
            StreamingOMSerializer serializer = new StreamingOMSerializer();
            reader =  StAXUtils.createXMLStreamReader(this.data);
           
            serializer.serialize(reader, xmlWriter);// OutOfMemory when 
MTOM enabled because output does not stream. Bug is filed and is being fixed
        } catch (Throwable e)
        {
                InitializationServlet.error("", e);
        }
    }
//need to implement other methods for interface but above method gets 
called when OMElement returned from a service
}

Make sure MTOM is disabled for streaming to work do to a bug that is 
currently being fixed.

Hope this helps someone.

Thanks,

Tammy

Tammy Dugan wrote:
> After doing some more testing, it looks like my solution works. 
> Before, I never got an out of memory message, but the returned xml 
> file was empty for a long time (20 minutes or more even though content 
> was being written to it the whole time). I never let the process 
> finish, though. I guess a larger amount than I expected was buffered 
> before being written. I received no out of memory errors on a 46MB 
> file so I guess everything is fine. I will do a little more testing to 
> confirm this.
>
> Tammy
>
> Davanum Srinivas wrote:
>> Ah. Sorry. need to think about that
>>
>> On 6/28/07, Tammy Dugan <tdugan@regenstrief.org> wrote:
>>> I'm not streaming attachments. I am streaming soap body xml. Will the
>>> custom dataHandler work for that too?
>>>
>>> Thanks,
>>>
>>> Tammy
>>>
>>> Davanum Srinivas wrote:
>>> > http://marc.info/?l=axis-user&m=118282183314980&w=2
>>> >
>>> > On 6/28/07, Tammy Dugan <tdugan@regenstrief.org> wrote:
>>> >> With a little work, I was able to get streaming soap body xml output
>>> >> with input coming from a database to work for very large files 
>>> (200MB).
>>> >> Thanks, Ryan. However, now I want to be able to stream dynamically
>>> >> created xml. Here is most of the code from my service:
>>> >>
>>> >>         StringDOM dom = new StringDOM();
>>> >>         PipedOutputStream output = new PipedOutputStream();
>>> >>         final StringDocument variableListDoc =
>>> >> dom.createDocument(output);
>>> >>         InputStream inputStream = new PipedInputStream(output);
>>> >>         OMFactory factory = OMAbstractFactory.getOMFactory();
>>> >>         OMElement resultElement =
>>> >> 
>>> XMLUtil.parseXMLFromStreamOM(inputStream,"getQueryVariableListResponse",true,factory);

>>>
>>> >>
>>> >>         final StringElement resultXML = (StringElement)
>>> >> variableListDoc.createElement(RESPONSE_XML_NODENAME);
>>> >>         Thread thread = new Thread(new Runnable()
>>> >>                 {
>>> >>                         public void run()
>>> >>                         {
>>> >>                                 try
>>> >>                                 {
>>> >>                                         Connection con =
>>> >> db.openConnection();
>>> >>                                         StringElement result =
>>> >> (StringElement) qdb.readQueryVariableList(con, db, variableListDoc);
>>> >>                                         
>>> resultXML.appendChild(result);
>>> >>                                         result.closeBuffer();
>>> >>                                 } catch (Exception e)
>>> >>                                 {
>>> >>
>>> >> InitializationServlet.error("", e);
>>> >>                                 }
>>> >>                         }
>>> >>                 });
>>> >>
>>> >>                 thread.start();
>>> >>
>>> >>                 return resultElement;
>>> >>
>>> >>
>>> >> StringDom is a DOM implementation one of our developers wrote that
>>> >> streams the content of a node added through an appendChild call, 
>>> thus
>>> >> very little is ever held in memory.
>>> >> When the above service runs, it is buffering the content of
>>> >> resultElement in memory and not streaming it to the response. I
>>> >> copied the exact same code to a local test case and ran
>>> >> serializeAndConsume on resultElement.
>>> >> When I did this, everything streamed just fine. I am using a Custom
>>> >> OMDataSource to create resultElement as an OMSourcedElementImpl to
>>> >> facilitate the streaming. When I ran the local example that 
>>> streamed,
>>> >> the following method was called in my custom data source:
>>> >>
>>> >> public void serialize(OutputStream output, OMOutputFormat format)
>>> >> throws XMLStreamException
>>> >>         {
>>> >>                 System.out.println("CustomDataSource
>>> >> serialized(output,format)");
>>> >>                 try
>>> >>                 {
>>> >>
>>> >> org.regenstrief.util.Util.bufferedReadWrite(this.data, output, 
>>> false);
>>> >>                 } catch (IOException e)
>>> >>                 {
>>> >>                         throw new XMLStreamException(e);
>>> >>                 }
>>> >>         }
>>> >>
>>> >> where bufferedReadWrite just reads the bytes from an input stream 
>>> and
>>> >> writes them to an output stream in chunks. The serialization method
>>> >> called when I executed the service was the following:
>>> >>
>>> >> public void serialize(XMLStreamWriter xmlWriter) throws
>>> >> XMLStreamException
>>> >>         {
>>> >>                 System.out.println("CustomDataSource
>>> >> serialized(xmlWriter)");
>>> >>                 XMLStreamReader reader = null;
>>> >>
>>> >>                 try
>>> >>                 {
>>> >>                         StreamingOMSerializer serializer = new
>>> >> StreamingOMSerializer();
>>> >>                         reader =
>>> >> StAXUtils.createXMLStreamReader(this.data);
>>> >>                         serializer.serialize(reader, xmlWriter);//
>>> >> OutOfMemory error here
>>> >>                 } catch (Throwable e)
>>> >>                 {
>>> >>                                 InitializationServlet.error("", e);
>>> >>                 }
>>> >>         }
>>> >>
>>> >> This method would not stream the data to the output. Is there
>>> >> something about StreamingOMSerializer that it is hanging while the
>>> >> xml is getting dynamically built? This same method works just 
>>> fine with
>>> >> an InputStream from a database so why doesn't it work for a
>>> >> PipedInputStream?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Tammy
>>> >>
>>> >>
>>> >> >I've had some luck getting this to work by implementing an
>>> >> OMDataSource that
>>> >> >doesn't start writing XML until it is passed a writer and then 
>>> using
>>> >> it to
>>> >> >create an OMSourcedElementImpl.  It sort of depends on the 
>>> semantics
>>> >> of what
>>> >> >you are doing, and you have to be careful as to what methods get
>>> >> called on
>>> >> >the OMSourcedElementImpl (it doesn't seem to take much to cause
the
>>> >> whole
>>> >> >thing to inflate - try to avoid doing anything that causes the
>>> >> getReader()
>>> >> >method to get called before you are ready to start sending stuff

>>> out
>>> >> the
>>> >> >door), but we've managed to get true streaming to work using
>>> >> Axis2/Axiom,
>>> >> >which saves a lot of transient memory.
>>> >>
>>> >> --
>>> >> Tammy Dugan
>>> >> Computer Programmer
>>> >>
>>> >> Regenstrief Institute, Inc.
>>> >> Medical Informatics
>>> >> Health Information and Translational Sciences (HITS) Building
>>> >> 410 West 10th Street, Suite 2000
>>> >> Indianapolis, IN 46202
>>> >> Main: 317.423.5500
>>> >> Fax: 317.423.5695
>>> >> IU campus mail address: HS, 2000
>>> >>
>>> >> (317) 423 - 5541
>>> >>
>>> >> Confidentiality Notice: The contents of this message and any files
>>> >> transmitted with it may contain confidential and/or privileged
>>> >> information and are intended solely for the use of the named
>>> >> addressee(s). Additionally, the information contained herein may 
>>> have
>>> >> been disclosed to you from medical records with confidentiality
>>> >> protected by federal and state laws. Federal regulations and State
>>> >> laws prohibit you from making further disclosure of such information
>>> >> without the specific written consent of the person to whom the
>>> >> information pertains or as otherwise permitted by such 
>>> regulations. A
>>> >> general authorization for the release of medical or other 
>>> information
>>> >> is not sufficient for this purpose.
>>> >>
>>> >> If you have received this message in error, please notify the sender
>>> >> by return e-mail and delete the original message. Any retention,
>>> >> disclosure, copying, distribution or use of this information by
>>> >> anyone other than the intended recipient is strictly prohibited.
>>> >>
>>> >>
>>> >>
>>> >> 
>>> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>>> >> For additional commands, e-mail: axis-user-help@ws.apache.org
>>> >>
>>> >>
>>> >
>>> >
>>>
>>> -- 
>>> Tammy Dugan
>>> Computer Programmer
>>>
>>> Regenstrief Institute, Inc.
>>> Medical Informatics
>>> Health Information and Translational Sciences (HITS) Building
>>> 410 West 10th Street, Suite 2000
>>> Indianapolis, IN 46202
>>> Main: 317.423.5500
>>> Fax: 317.423.5695
>>> IU campus mail address: HS, 2000
>>>
>>> (317) 423 - 5541
>>>
>>> Confidentiality Notice: The contents of this message and any files 
>>> transmitted with it may contain confidential and/or privileged 
>>> information and are intended solely for the use of the named 
>>> addressee(s). Additionally, the information contained herein may 
>>> have been disclosed to you from medical records with confidentiality 
>>> protected by federal and state laws. Federal regulations and State 
>>> laws prohibit you from making further disclosure of such information 
>>> without the specific written consent of the person to whom the 
>>> information pertains or as otherwise permitted by such regulations. 
>>> A general authorization for the release of medical or other 
>>> information is not sufficient for this purpose.
>>>
>>> If you have received this message in error, please notify the sender 
>>> by return e-mail and delete the original message. Any retention, 
>>> disclosure, copying, distribution or use of this information by 
>>> anyone other than the intended recipient is strictly prohibited.
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: axis-user-help@ws.apache.org
>>>
>>>
>>
>>
>

-- 
Tammy Dugan
Computer Programmer

Regenstrief Institute, Inc.
Medical Informatics
Health Information and Translational Sciences (HITS) Building
410 West 10th Street, Suite 2000
Indianapolis, IN 46202
Main: 317.423.5500
Fax: 317.423.5695
IU campus mail address: HS, 2000

(317) 423 - 5541

Confidentiality Notice: The contents of this message and any files transmitted with it may
contain confidential and/or privileged information and are intended solely for the use of
the named addressee(s). Additionally, the information contained herein may have been disclosed
to you from medical records with confidentiality protected by federal and state laws. Federal
regulations and State laws prohibit you from making further disclosure of such information
without the specific written consent of the person to whom the information pertains or as
otherwise permitted by such regulations. A general authorization for the release of medical
or other information is not sufficient for this purpose.
 
If you have received this message in error, please notify the sender by return e-mail and
delete the original message. Any retention, disclosure, copying, distribution or use of this
information by anyone other than the intended recipient is strictly prohibited.



---------------------------------------------------------------------
To unsubscribe, e-mail: axis-user-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-user-help@ws.apache.org


Mime
View raw message