camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Baris Acar <ba...@acar.org.uk>
Subject Robustly streaming messages to archive
Date Fri, 18 Oct 2013 10:25:12 GMT
Hi,
We use camel in a standalone app which responds to a user request by sending some messages
via JMS. To aid with troubleshooting/support, we also save these messages to disk using the
'file:' endpoint. 

Unfortunately, since there are many tens of thousands of smallish (5-10KiB) messages per user
request, the folders are quite unwieldy to deal with. e.g. transferring the saved messages
from a network share is particularly slow compared to transferring a single larger file.

We tried aggregating all such messages for a user request and creating a zip archive on completion;
however this has the drawback that nothing is logged if the process crashes or is forcibly
terminated partway through a request.

Is there a camel-y solution to this problem? At the moment we are thinking of writing the
messages into a single file via a tar stream, and possibly subsequently a gzip stream. My
understanding is that, unlike .zip, both .tar and .gz support streaming (i.e. can tolerate
partially complete archives), although I'm not an expert. I was wondering how the community
has solved this problem?

Thanks,
Barış
Mime
View raw message