camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zoran Regvart <zo...@regvart.com>
Subject Re: Best way to poll a database and process in chunks
Date Thu, 06 Apr 2017 22:43:11 GMT
Hi David,

I think you will find that binding a bigger number of parameters won't
lead to better performance, both with missing statement cache due to
fluctuating number of parameters and due to not using indexes and
performing full table scans. But you should run SQL explain and gather
statistics for your own use case to see that.

I also think that you should solve this at the SQL level first. For
instance select first n rows from your table with certain status and
process those as a batch.

zoran

On Thu, Apr 6, 2017 at 6:59 PM, dvlato <dvlato@gmail.com> wrote:
> Hello Claus, Ibsen:
>
>  Thank you very much for the thorough responses. We don't really need to use
> the JPA component, and I think it makes more sense to use sql/jdbc for the
> kind of bulk operations we want to do.
>
> What I think it would be really useful is to allow for the
> "onConsumeBatchComplete" SQL statement  to use parameters from the  select
> query.  I mean, if the sql consumer is performing the query  "select * from
> mytable where status=0",  I would like to be able to declare  the
> "onConsumeBatchComplete" query like this " update mytable set status=1 where
> idfield in (:idfield)", where the idfield parameter would be populated with
> the results of the select statement.
>
>  Does my explanation make sense? I want to change the status only for the
> rows returned by the select query.
>
>  Is there another way to do that currently in Camel?  I tried to connect the
> output of the SQL component with another SQL component but I think I might
> need a bean processor to generate the correct body. In any case, I think
> that would be a useful built-in feature, so if you agree I will create a
> ticket for that.

-- 
Zoran Regvart

Mime
View raw message