axis-c-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Senaka Fernando" <sen...@wso2.com>
Subject Re: Caching support for large attachments
Date Sat, 15 Mar 2008 08:09:30 GMT
>>>  BTW, this whole discussion is about in path, that is reading an
>>>  incomming message. How about the out path? We have the same problems
>>>  when sending attachments. Right now, we read the whole file into
>>> memory
>>>  and then only we send over the wire.
>> hmm... Why not write it in chunks.. Read a chunk from the file, then
>> write it to the outstream.. Use size of the file for content-type
>> calculation in case of non-chunking.. But mostly people will use
>> chunking when using MTOM..
>
> No, chunking is not required. You also don't need to write the entire data
> to be sent, to the stream at once. Because any HTTP Receiver will pull
> from the stream until it sees a valid ending character sequence.

It should rather read a length equal to content length. And the
terminating sequence is for headers. Sorry for the confusion. Therefore,
the HTTP Receiver will pull from the stream until it reads a content
length or until an error occurs.

>
> I believe that you should be able to write part by part to the stream, and
> send it, then reuse the buffer and write part 2, and send and so on. This
> argument can be justified, because on the receiving end, we must read the
> multi-part data until we encounter the mime boundary, unlike an ordinary
> payload where it can be terminated by a valid terminating character

Same here. We'll be reading a length equal to content length.

> sequence . We'll only have issues if we are to write large soap payloads
> which of course can be dealt with once we've implemented Session in
> Axis2/C.
>
> Regards,
> Senaka
>
>>
>> thanks,
>> Thilina
>>
>>
>>>
>>>  Samisa...
>>>
>>>
>>>
>>>  > Regards,
>>>  > Senaka
>>>  >
>>>  >
>>>  >> Hi,
>>>  >>
>>>  >>>  > In Axis2/Java case we do write the attachment content directly
>>> from
>>>  >>>  > the InputStream to the File when the attachment size is larger
>>> than
>>>  >>>  > the threshold.  This avoids loading the whole attachment
to the
>>>  >>> memory
>>>  >>>  > at all.
>>>  >>>
>>>  >>>  In this case to find out the attachment size don't you need to
do
>>> any
>>>  >>>  mime parsing? How do you find the attachment size with out
>>> searching
>>>  >>> for
>>>  >>>  the mime boundaries ?
>>>  >>>
>>>  >> Yes.. MIME is a boundary based packaging mechanism and you does not
>>>  >> need to specify the length for each of the parts...Even the HTTP
>>>  >> content length is not there if the message is chunked.
>>>  >>
>>>  >> What we did in Axis2/Java to overcome this is to read the data to a
>>>  >> byte[] buffer of up to a certain size (the size threshold). If
>>> there
>>>  >> are more data available in the mime part (if we have not
>>> encountered
>>>  >> the boundary yet) then we know this attachment is bigger than the
>>>  >> threshold. So we create the temp file, pump the content in the
>>> buffer
>>>  >> to the file, then pump the rest of the stream to the file.. In this
>>>  >> way we do not need to know the size of the attachment upfront.. BTW
>>> we
>>>  >> do all of the above while we are parsing the MIME message at the
>>> MIME
>>>  >> parser level..
>>>  >>
>>>  >>
>>>  >>>  > This has the plus point that the attachment size will be
>>>  >>>  > limited only by the available free space in the Temp
>>> Directory..
>>>  >>>  > Will that be possible in Axis2/C.. Or is that wat you have
in
>>> mind
>>>  >>> :)..
>>>  >>>
>>>  >>>  Yes this is possible.
>>>  >>>
>>>  >> But in Axis2/JAVA we will get a OutOfMemory if we parse a large
>>> MIME
>>>  >> part upfront, since it reads the attachment to memory. May be you
>>> can
>>>  >> have a larger limit with C than in Java, but ultimately you'll come
>>> to
>>>  >> a situation where you will not have enough memory to store that
>>> MIME
>>>  >> part in memory in the parsing time, unless you write in to a File
>>>  >> while parsing,..
>>>  >>
>>>  >> thanks,
>>>  >> Thilina
>>>  >>
>>>  >>
>>>  >>>
>>>  >>>  >
>>>  >>>  > thanks,
>>>  >>>  > Thilina
>>>  >>>  >
>>>  >>>  >  >and keeping the file name inside
>>>  >>>  > >  data_handler instead of the whole buffer. So the service
or
>>> the
>>>  >>> client
>>>  >>>  > >  will get the file name instead of the buffered stream,
when
>>> it
>>>  >>> receives
>>>  >>>  > >  an attachment. This will not prevent buffering the
>>> attachment
>>> at
>>>  >>> the
>>>  >>>  > >  transport but will prevent keeping it inside the om_tree
>>> till
>>> it
>>>  >>> reaches
>>>  >>>  > >  the receiver.
>>>  >>>  > >
>>>  >>>  > >  Before implementing this I would like to know your
>>> suggestions
>>>  >>> regarding
>>>  >>>  > >  this.
>>>  >>>  > >
>>>  >>>  > >  [1] https://issues.apache.org/jira/browse/AXIS2C-672
>>>  >>>  > >
>>>  >>>  > >  Thanks,
>>>  >>>  > >  -Manjula
>>>  >>>  > >
>>>  >>>  > >  --
>>>  >>>  > >  Manjula Peiris: http://manjula-peiris.blogspot.com/
>>>  >>>  > >
>>>  >>>  > >
>>>  >>>  > >  ---------------------------------------------------------------------
>>>  >>>  > >  To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>  >>>  > >  For additional commands, e-mail:
>>> axis-c-dev-help@ws.apache.org
>>>  >>>  > >
>>>  >>>  > >
>>>  >>>  >
>>>  >>>  >
>>>  >>>  >
>>>  >>>
>>>  >>>
>>>  >>>  ---------------------------------------------------------------------
>>>  >>>  To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>  >>>  For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>  >>>
>>>  >>>
>>>  >>>
>>>  >>
>>>  >> --
>>>  >> Thilina Gunarathne - http://thilinag.blogspot.com
>>>  >>
>>>  >> ---------------------------------------------------------------------
>>>  >> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>  >> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>  >>
>>>  >>
>>>  >>
>>>  >
>>>  >
>>>  > ---------------------------------------------------------------------
>>>  > To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>  > For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>  >
>>>  >
>>>  >
>>>  >
>>>
>>>
>>>  --
>>>  Samisa Abeysinghe
>>>  Software Architect; WSO2 Inc.
>>>
>>>  http://www.wso2.com/ - "Oxygenating the Web Service Platform."
>>>
>>>
>>>
>>>
>>>  ---------------------------------------------------------------------
>>>  To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>>>  For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>>
>>>
>>
>>
>>
>> --
>> Thilina Gunarathne - http://thilinag.blogspot.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org


Mime
View raw message