cxf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Diephouse <dan.diepho...@mulesource.com>
Subject Re: Large messages and memory thoughts.....
Date Wed, 10 Oct 2007 17:38:02 GMT
Daniel Kulp wrote:
> I did quite a bit of work yesterday to fix some areas where large 
> messages were causing problems.   They all pretty much centered around 
> mis-use of the CachedOutputStream (assuming it's always a 
> ByteArrayOutputStream which isn't true once it flips to a file).   
> However, while digging through those, I've noticed a few other potential 
> problems that I'd like to discuss.
>
> 1) Logging interceptors - right now, we log the entire message to the 
> Logger.   However, the logger requires a String, so the entire message 
> is converted to a String (keep in mind, a String is utf-16 so twice the 
> memory is consumed) for logging.   Plus, do you really think logging a 
> 10MB message to the Logger is the best option?  I'm going to add a 
> configurable "limit" that will Log just the first "limit" bytes of the 
> message.   Should the default be -1 (no limit) or something reasonable 
> like 4K?      Also, we could have a configuration to allow it to 
> optionally log to a separate file instead of the Logger.  We can then 
> directly copy from stream to stream with very little memory usage.  
> That's going to be a lot trickier though as we'll have to deal with 
> concurrent access to the file from multiple threads and such. 
>   
Maybe a more reasonable limit would be 100 K and a warning message 
otherwise.
> 2) JMS transport - the JMS transport always deals with the entire message 
> as either a String (for TextMessage) or byte[].   That seems to be a 
> limitation for standard JMS API's.   Some vendors (such as ActiveMQ) do 
> have proprietary extensions to allow some proprietary stuff to help.  
> Examples:
> http://activemq.apache.org/blob-messages.html
> http://activemq.apache.org/jms-streams.html
> It might be good to allow some extra hooks into the JMS transport to 
> allow for easy subclassing to enable some of the hooks.   (That said, 
> I'd like to see the JMS transport also changed over to using the Spring 
> JMS configuration.)
>
>   
+1 on both accounts
> 3) WS-RM - The RMMessage object holds the message as a byte[].   Thus, 
> it's entirely in memory until it's stored.  It would probably be a good 
> idea for this to be passed as the original CachedOutputStream and stored 
> directly to/from that stream
Yeah, I suppose we could stream it to its storage location and then 
stream it back out later on. Never thought about that, but that makes 
perfect sense now that you mention it!

- Dan

-- 
Dan Diephouse
MuleSource
http://mulesource.com | http://netzooid.com/blog


Mime
View raw message