camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billy <billy.sjob...@gmail.com>
Subject Need tips for splitter/aggregator with large data
Date Thu, 24 Nov 2011 14:47:14 GMT
Hi all!

I have a requirement to transform a fairly large set of xml data that I
fetch over HTTP and then FTP it out.
After getting my first OOME I started reading up a bit on how you could
utilize the splitter with streaming and tokenizing pairs (2.9RC) which
worked well for me for splitting and transforming the individual pieces.
(Great work btw on this!)

However, I have not found a good way to aggregate the data back again after
the transformation. My custom aggregationstrategy that I set on the splitter
simply concatinates the payloads into either a String or byte[], but this
also gives me memory problems.

Now, the streamCaching on the route coupled with a streaming splitter solves
the splitting part, but can anyone hint me in the right direction of how I
should effectively piece the parts back during the aggregation phase?
Basically offloading the storage of the aggregated data from memory..

I have played around with the concept of simply appending all
split/transformed elements to a file, and ftp that out after it has been
marked complete. This works, but then I loose all my state inside the route
which I need for later reporting.

--
View this message in context: http://camel.465427.n5.nabble.com/Need-tips-for-splitter-aggregator-with-large-data-tp5020356p5020356.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message