camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dvlato <>
Subject Best way to poll a database and process in chunks
Date Fri, 24 Mar 2017 12:02:46 GMT
I have just started working with Camel and I have what I believe is a pretty
common requirement:  poll the database to fetch all the records with a
certain status, and process them in chunks (let's 10 rows at a time). I
though I could leverage the existing components to do this easily but I am
not finding the correct way:

What I have tried so far is to usa the JPA component (as we already have JPA
entities in place) to poll the database and use the @Consumed annotation to
change the status of the record (some pointers about how to deal with
transactionality are also most welcome).  The problem with this approach is:
1) Here the rows are received one by one, and we can later merge them with
an aggregator, which seems suboptimal.
2)  Even if we ignored the performance problem with that approach, I'm not
sure of how to configure the aggregator to have fixed size blocks but taking
into account the whole size of the batch (I mean, if after chunking we only
have 5 rows in a message, just return that instead of waiting for the 10th -
maybe it works like that by default - I don't really know).

I've seen that there is a class called  "JpaPollingConsumer" which returns
all the rows instead of one by one, but I don't know if there is anyway to
use it with a "from()" clause, it seems that Camel calls "createConsumer"
and not "createPollingConsumer", right? Is there a way to use
from("jpa://...") and have Camel use the polling consumer?  I have tried
using pollEnrich but in that case the @Consumed code is not executed. Is
there any other alternative?

 I apologise for such basic questions, I am sure I could probably figure out
a "good enough" approach using my inexistent knowledge of Camel (we have
Camel in Action which has been great so far, but I don't see anything in
Chapter 6 regarding database polling) but, on the other hand, it's such a
common task that someone might have best practices  already available.  I
have browsed the myBatis and SQL docs and I think that they might be better
suited (if I am not using JPA incorrectly, which is quite likely the case),
but I will have the same problem of not knowing if I am using Camel

View this message in context:
Sent from the Camel - Users mailing list archive at

View raw message