cxf-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Kulp <dk...@apache.org>
Subject Re: Streaming large attachments by linking input and output stream avoiding temp files
Date Thu, 17 Sep 2009 02:32:45 GMT
On Wed September 16 2009 12:31:57 pm K1001010 wrote:
> Did these changes make it into the 2.2.3 release?

It looks like it, yea.


> I am testing with a 150MB attachment, and still see it being streamed to
> disk

Hmm...   I hate to ask this as I don't want a 150MB test case, but is there 
any way you could create a "small" test case?

There are a few cases where we do have to buffer it ahead of time.   If there 
are any SAAJ related things, we need to buffer.  Schema validation is another.  
If there are multiple attachments, it's likely the others may need to be 
buffered, etc...   A testcase would help figure out what is going on.

If you can debug it by setting a breakpoint in 
org.apache.cxf.attachment.AttachementDataSource.cache() and determine why 
cache is being called, that would also be helpful.

Dan



> 
> I have this code:
> 
> 
> Picture picture = serviceProxy.getPictuer();
> DataHandler handler = picture.getImageData();
> InputStream is = handler.getInputStream();
> 
> 
> In version 2.2.2 the entire 150BM file would be streamed to temp file on
> line 3, which would return a file input stream (or a delegate too).
> 
> In version 2.2.3 the entire 150MB file is streamed to temp file before line
> 1 returns.
> 
> I disable schema validation using this system property:
> 
> org.apache.spring.validation.mode=VALIDATION_NONE
> 
> Is that correct?   Anything else required to stream directly from the wire?
> 
> Thanks for any help.
> 
> dkulp wrote:
> > Well, I THOUGHT this would have been easily doable, but when I checked,
> > it turns out the attachments were not streaming properly in MTOM mode. 
> > :-(
> >
> > I've gone through and fixed a BUNCH of things related to the attachment
> > handling and streaming and such so some of this is now possible.
> >
> > Now, with the latest code, when you get the DataSource, if at all
> > possible,
> > the input stream that is returned will properly be streaming off the wire
> > instead of cached.   Thus, you can copy directly from there to the output
> > stream.
> >
> > There are cases where this WON'T work stream directly, notably when
> > schema validation is turned on.   In that case, to work around issues in
> > JAXB, we buffer the content.     Also, if there are multiple attachments,
> > only the last
> > one can really be streamed this way.   The others get buffered.
> >
> > Dan
> >
> > On Mon July 27 2009 5:50:00 am dp-isp wrote:
> >> Hello everybody,
> >>
> >> We have a CXF webservice that is a kind of "repository service" and a
> >> webapp working as frontend for that repository.
> >> Those two components are on separate hosts.
> >>
> >> We would need to realize a continuous stream of data when downloading
> >> attachments from CXF webservice to the users' browser through the
> >> frontend
> >> webapp.
> >>
> >> The webapp is basically a servlet that invokes the webservice, receives
> >> the
> >> attachment datahandler and copies datahandler InputStream to servlet
> >> response OutputStream (servlet code is at the end of this email).
> >>
> >> Our issue: is it possible to directly link datahandler InputStream to
> >> servlet response OutputStream without re-building the whole attachment
> >> in memoroy or in a temporary file? (as can be configured by setting
> >> org.apache.cxf.io.CachedOutputStream.Threshold,
> >> org.apache.cxf.io.CachedOutputStream.OutputDirectory)
> >>
> >> Going more into details, is there a way to send to CXF the output buffer
> >> to
> >> write the attachment on? In such case we would send to CXF the sevlet's
> >> output buffer.
> >>
> >> Alternatively, is there a way to manage CXF working asynchronously with
> >> attachments, giving back control to the servlet before attachment
> >> downolad
> >> completion?
> >>
> >> Many thanks,
> >> Marco
> >>
> >> ---
> >>
> >> This is the servlet's involved code:
> >>
> >> ...
> >> StreamingService_Service ss = new StreamingService_Service(new
> >> URL(serviceURL));
> >> StreamingService port = ss.getServiceImplPort(new MTOMFeature());
> >>
> >> Resume myFile = port.downloadMethod(fileName, fileType, usr, pwd, repo);
> >> DataHandler handler = myFile.getResume();
> >>
> >> response.setHeader("Pragma", "No-cache");
> >> response.setContentType("application/octet-stream");
> >> response.setHeader("content-disposition", "attachment; filename=" +
> >> fileName + "." + fileType);
> >> try {
> >> 	InputStream is = handler.getInputStream();
> >> 	try {
> >> 		OutputStream os = response.getOutputStream();
> >> 		try {
> >> 			byte[] b = new byte[100000];
> >> 			int bytesRead = 0;
> >> 			while ((bytesRead = is.read(b)) != -1) {
> >> 				os.write(b, 0, bytesRead);
> >> 			}
> >> 			System.out.println("Download del file " + myFile.getCandidateName() +
> >> "." + myFile.getResumeFileType() + " completato.");
> >> 		} finally {
> >> 			os.flush();
> >> 			os.close();
> >> 		}
> >> 	} finally {
> >> 		is.close();
> >> 	}
> >>
> >> } catch (Exception e) {
> >> 	e.printStackTrace();
> >> }
> >> ...
> 

-- 
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog

Mime
View raw message