camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <>
Subject Re: Streaming huge files in Camel
Date Fri, 11 Apr 2014 08:37:40 GMT

Maybe poll from the database with a limited set of data per poll.

Also is 1 row in the database = 1 xml
Then you can just poll one row at a time, not as efficient but it gets
the job done.

On Fri, Apr 11, 2014 at 9:58 AM, davidrobert <> wrote:
> We have the following requirement:
> 1. Poll from a database
> 2. Marshall the result set into XML
> 3. Send the XML in a JMS message to a downstream system
> The issue is, some of the result sets could be huge - millions of rows. The
> transformer may be unable to handle this simply because of memory/heap space
> limitations. There is also the issue of sending a massive message.
> Does Camel have an easy solution for this, some way of streaming the result
> set to a transformer in small chunks, then sending individual messages for
> each piece which are identified as part of a collection by some kind of
> correlation id or something similar?
> Or is our requirement simply not suited for messaging infrastructure, in
> which case we should look at a solution based around batch processing?
> Thanks
> --
> View this message in context:
> Sent from the Camel - Users mailing list archive at

Claus Ibsen
Red Hat, Inc.
Twitter: davsclaus
Author of Camel in Action:

View raw message