Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (athena.apache.org: domain of claus.ibsen@gmail.com
 designates 209.85.223.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1397203111357-5750096.post@n5.nabble.com>
References: <1397203111357-5750096.post@n5.nabble.com>
From: Claus Ibsen <claus.ibsen@gmail.com>
Date: Fri, 11 Apr 2014 10:37:40 +0200
Message-ID: 
 <CAGB5yNnX8D3eL9EVVNSxMeruGrPWHoQgU2RFQc_4PmS0xXqXWA@mail.gmail.com>
Subject: Re: Streaming huge files in Camel
To: "users@camel.apache.org" <users@camel.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Hi

Maybe poll from the database with a limited set of data per poll.

Also is 1 row in the database = 1 xml
Then you can just poll one row at a time, not as efficient but it gets
the job done.


On Fri, Apr 11, 2014 at 9:58 AM, davidrobert <davidrfield@gmail.com> wrote:
> We have the following requirement:
> 1. Poll from a database
> 2. Marshall the result set into XML
> 3. Send the XML in a JMS message to a downstream system
>
> The issue is, some of the result sets could be huge - millions of rows. The
> transformer may be unable to handle this simply because of memory/heap space
> limitations. There is also the issue of sending a massive message.
>
> Does Camel have an easy solution for this, some way of streaming the result
> set to a transformer in small chunks, then sending individual messages for
> each piece which are identified as part of a collection by some kind of
> correlation id or something similar?
>
> Or is our requirement simply not suited for messaging infrastructure, in
> which case we should look at a solution based around batch processing?
>
> Thanks
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096.html
> Sent from the Camel - Users mailing list archive at Nabble.com.


-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/