axis-java-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davanum Srinivas <dava...@gmail.com>
Subject Re: [Axis2] File Caching for Attachments
Date Mon, 15 Aug 2005 15:23:47 GMT
Now we need to make sure that they get cleaned up "auto-magically" as well :)

-- dims

On 8/15/05, Tom Jordahl <tomj@macromedia.com> wrote:
>  
>  
> 
> Sweet – Thanks Thilina! 
> 
>   
>  
> 
> --
>  Tom Jordahl
>  Macromedia Server Development 
>  
>  
>  ________________________________
>  
> 
> From: Thilina Gunarathne [mailto:csethil@gmail.com] 
>  Sent: Friday, August 12, 2005 1:13 AM
>  To: dims@apache.org
>  Cc: axis-dev@ws.apache.org
>  Subject: Re: [Axis2] File Caching for Attachments 
> 
>   
> 
> DONE .... File caching based on the size of the attachment is done......
>  We buffer the attachements to see whether it exceeds the given limit. If it
> is not exceeded simply create a memory part using the buffer.. If it exceeds
> the limit then the buffer is fulshed to a file and the remaining data in the
> attachments is directly pipied to the file after that...
>  
>  Thanks,
>  ~Thilina 
>  
> 
> On 8/8/05, Davanum Srinivas <davanum@gmail.com> wrote: 
> 
> Sure Thilina. Just make sure there is a JIRA issue and we track it
>  appropriately so that it does not get forgotten.
>  
>  thanks,
>  dims
>  
>  On 8/8/05, Thilina Gunarathne <csethil@gmail.com > wrote:
>  > Hi,
>  >  In the current impl we have two Part implementations called MemoryPart
> and
>  > FilePart, which takes care of in memory buffering and file caching....
>  >   When it comes to in memory impl we use Java Mail to parse the Mime
> parts ( 
>  > we just feed the boundary delimited stream to MimePart in JavaMail and it
>  > handles parsing headers and creating Datahandlers internally..).
>  >   But when it comes to FileParts we parse the MimeParts( parsing the
> headers 
>  > and directing the rest of the stream to a temporary file) by our selves.
>  >
>  >  We can use this byte array strategy if we can parse the Memory parts
> also
>  > by our selves (yes we are slowly getting rid of JavaMail) .... This
> involves 
>  > bit of a risk and some work... So shall we differ this after to  the
> .91...
>  >
>  >  Thanks & Regards,
>  >  ~Thilina
>  >
>  >
>  >
>  > On 8/6/05, Davanum Srinivas < davanum@gmail.com> wrote:
>  > > Thilina,
>  > >
>  > > when the byte array grows beyond a certain size, dynamically you
>  > > switch to file storage on the fly....
>  > >
>  > > -- dims 
>  > >
>  > > On 8/6/05, Thilina Gunarathne <csethil@gmail.com > wrote:
>  > > > Yes.. We can .. But *most* of the time Mime parts does not specify a
>  > > > content-length. So is it worth the hassle??? 
>  > > >
>  > > > Thanx & Regards,
>  > > > ~Thilina
>  > > >
>  > > > On 8/6/05, Tom Jordahl < tomj@macromedia.com> wrote:
>  > > > >
>  > > > >
>  > > > >
>  > > > > Can you try to switch based on Content-Length?  If its not there,
> fall
>  > > > back to on/off behavior?
>  > > > > 
>  > > > >
>  > > > >
>  > > > >
>  > > > > Tom Jordahl
>  > > > > Macromedia Server Development
>  > > > >
>  > > > > ________________________________ 
>  > > >
>  > > > >
>  > > > > From: Thilina Gunarathne [mailto: csethil@gmail.com]
>  > > > > Sent: Saturday, August 06, 2005 9:09 AM
>  > > > > To: Tom Jordahl
>  > > > >
>  > > > > Subject: Re: [Axis2] File Caching for Attachments
>  > > > >
>  > > > >
>  > > > >
>  > > > > 
>  > > > >
>  > > > >
>  > > > >
>  > > > >
>  > > > >
>  > > > > A quick comment....
>  > > > >
>  > > > >
>  > > > > 
>  > > > > It seems it's not possible to have a size threshold according to
> how
>  > we
>  > > > are handling the file storage in Axis2.
>  > > > >
>  > > > >
>  > > > > We directly stream the incoming binary data to files after reading

>  > mime
>  > > > part headers..We are not reading and buffering them to memory. This
>  > gives no
>  > > > room to calculate sizes before writing to the disk. Currently what we
>  > are
>  > > > doing is if "file caching" is switched "on" all the parts will be 
>  > directed
>  > > > to files irrespective of sizes....I'll try to come up with a work
>  > around.
>  > > > >
>  > > > >
>  > > > >
>  > > > >
>  > > > > 
>  > > > > Any ideas or work arounds to solve this clean up problem are
> welcome.
>  > > > >
>  > > > >
>  > > > >
>  > > > > Thanx & regards,
>  > > > > 
>  > > > >
>  > > > > ~Thilina
>  > > > >
>  > > > >
>  > > > >
>  > > > > On 8/4/05, Tom Jordahl < tomj@macromedia.com > wrote:
>  > > > >
>  > > > >
>  > > > > +1 to solving the cleanup problem.
>  > > > >
>  > > > > It would also be great to have a size threshold - anything over
X 
>  > > > > K-bytes will be streamed to disk, otherwise we handle it in memory.
>  A
>  > > > > reasonable default should be used and of course it should be user
>  > > > > configurable.
>  > > > >
>  > > > >
>  > > > > Tom Jordahl
>  > > > > Macromedia Server Development
>  > > > >
>  > > > > -----Original Message-----
>  > > > > From: Davanum Srinivas [mailto: davanum@gmail.com]
>  > > > > Sent: Thursday, July 28, 2005 6:41 PM
>  > > > > To: axis-dev@ws.apache.org ; Thilina Gunarathne 
>  > > > > Subject: Re: [Axis2] File Caching for Attachments
>  > > > >
>  > > > > Thilina,
>  > > > >
>  > > > > Check Axis 1.X, see how the temp directory is used to store 
>  > > > > attachments when the working directory is not specified. Also in
a
>  > > > > servlet environment we should not clutter the bin directory with
>  > > > > attachments.
>  > > > > 
>  > > > > Not cleaning up attachments has been a *****LONG***** standing
> problem
>  > > > > with Axis 1.X, So PLEASE think of a way to do it and API to do it
>  > > > > explicitly (both on client and server). 
>  > > > >
>  > > > > thanks,
>  > > > > dims
>  > > > >
>  > > > > On 7/28/05, Thilina Gunarathne < csethil@gmail.com > wrote:
>  > > > > > Hi all,
>  > > > > > I implemented a  temporary file caching solution for Axis2.With
> this
>  > > > > > we'll be able to directly pump the mime parts in incoming stream
> to 
>  > > > > > temporary files, without buffering them at all in memory. We
> achieve
>  > > > > > this by parsing the part headers by our selves and then writing
> the
>  > > > > > rest of the part to a temporary file till we reach the
> boundary... 
>  > > > > >
>  > > > > > Anybody can switch on file caching from the server.xml. When
it's
>  > set
>  > > > > > to true all the parts will get stored as files.. Also one can

>  > specify
>  > > > > > the directory where attachments should be stored..
>  > > > > > When it's not mentioned we are saving it in the working directory
>  > > > > > which is bit troublesome.... 
>  > > > > > Also is there any way to delete the temporary files after
> consuming
>  > > > > them..
>  > > > > >
>  > > > > > thanx & regards,
>  > > > > > ~Thilina 
>  > > > > >
>  > > > > > --
>  > > > > > "May the SourcE be with u"
>  > > > > > http://www.bloglines.com/blog/thilina 
>  > > > > >
>  > > > >
>  > > > >
>  > > > > --
>  > > > > Davanum Srinivas -http://blogs.cocoondev.org/dims/
>  > > > >
>  > > > >
>  > > > >
>  > > > >
>  > > > > --
>  > > > > "May the SourcE be with u"
>  > > > > http://www.bloglines.com/blog/thilina
>  > > >
>  > > >
>  > > >
>  > > > --
>  > > > "May the SourcE be with u"
>  > > > http://www.bloglines.com/blog/thilina
>  > >
>  > >
>  > > --
>  > > Davanum Srinivas - http://blogs.cocoondev.org/dims/
>  > >
>  >
>  > 
>  >
>  > --
>  >
>  > "May the SourcE be with u"
>  > http://www.bloglines.com/blog/thilina
>  
>  
>  --
>  Davanum Srinivas - http://blogs.cocoondev.org/dims/ 
> 
> 
>  
>  
>  -- 
>  "May the SourcE be with u" 
>  http://www.bloglines.com/blog/thilina 


-- 
Davanum Srinivas : http://wso2.com/ - Oxygenating The Web Service Platform
Mime
View raw message