camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen" <claus.ib...@gmail.com>
Subject Re: Ability to process multiple files (file endpoint) simultaneously in a single Application
Date Fri, 19 Dec 2008 06:03:10 GMT
On Fri, Dec 19, 2008 at 12:17 AM, jonathanq <jquail@abebooks.com> wrote:
>
> I have been trying to figure out if this is possible - and I may be missing
> the answer completely...or maybe it's just not possible..
>
> What I need to do is this:
>
> I am trying to create a generic process that will accept SQL queries (that
> get sent to the trigger directory on a scheduled basis with a CRON entry) -
> the queries will take a (relatively) long time to execute.  So if a second
> query is scheduled to run while the first is still running - I want the
> second to run in a different thread so they can run simultaneously.
>
> I noticed there is a "concurrentConsumers" property on a JMS endpoint.  Does
> that do essentially this process, however it is only for JMS?  Like you
> could set concurrentConsumers = 4 - and with a single application you can
> have it processing 4 JMS messages simultaneously?  (if it's a Queue - same
> message, 4 times if it is a Topic).
>
> This is basically the route that I am using for the process:
>
> from("file://c:/triggerdirectory").process(someProcessor);
>
> However this will (in my understanding) only handle 1 file at a time.  So if
> a second file arrives, it will not get processed until the first one is
> finished and returns from the processor.
Yes.

>
> Reading the concurrentConsumers documentation
> (http://activemq.apache.org/camel/competing-consumers.html) - I am wondering
> if something like this might work...
>
> from("file://c:/triggerdirectory").to("seda:filetrigger")
> from("seda:filetrigger?concurrentConsumers=5").to(someProcessor);
>
> Will that then cause the triggered file to go the seda queue and then have
> the second route do the parallel processing (up to 5 at a time) of the file
> while the first route will go back to listening to the directory?  Assuming
> this is all defined within a single camel context and running application.
Yeah you are nearly there. Splitting the route into from -> seda. seda
-> process is a good idea for parallelism.
However concurrentConsumers is an URI option only for JMS. You can use
the thread DSL instead

from("file://c:/triggerdirectory").to("seda:filetrigger")
from("seda:filetrigger").thread(5).to(someProcessor);

Some links:
http://activemq.apache.org/camel/competing-consumers.html
http://activemq.apache.org/camel/seda.html


>
> Any ideas?  Thanks in advance!
> --
> View this message in context: http://www.nabble.com/Ability-to-process-multiple-files-%28file-endpoint%29-simultaneously-in-a-single-Application-tp21082906s22882p21082906.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 

/Claus Ibsen
Apache Camel Committer
Blog: http://davsclaus.blogspot.com/

Mime
View raw message