camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gmagniez <gabriel.magn...@steria.com>
Subject Re: Processing 5 files at a time - Threads? SEDA + Concurrent Consumers?
Date Wed, 30 Dec 2009 10:51:43 GMT


jonathanq wrote:
> 
> I am trying to write a process that will use a file endpoint (camel 2.1.0)
> to read from a directory.
> 
> I need the process to read a file from the directory and then do some
> processing on the contents (namely hitting a REST service for each record
> in the file).  We have been asked to limit the number of threads that are
> hitting the service to 5.  So we decided to simply process 5 files at a
> time (to avoid concurrency issues reading 1 file and writing to 1 file
> with 5 threads)
> 
> I tried a few different approaches, and I wanted to see if there was a way
> to do what I want.
> 
> Approach 1:
> 
> from("file://incoming").to("seda:filequeue")
> 
> from("seda:filequeue").thread(5).process()
> 
> Now - this reads in ALL of the files in the directory (places camelLock on
> all) and then sends them to the seda endpoint.  I saw log messages that
> referred to thread 1 through 6.  But from what I read on the
> documentation, thread() is not necessarily going t limit it at that
> number. 
> 
> Approach 2:
> 
> from("file://incoming").thread(5).process()
> 
> This only processed 5 files at a time - but created camelLocks on all
> files in the directory.
> 
> So then I tried approach 3:
> 
> from("file://incoming").to("seda:filequeue")
> 
> from("seda:filequeue?concurrentConsumers=5").process()
> 
> Again this seems to work, however it puts a camelLock on all the files
> (because they were all processed by the first part of the route, they are
> just queued up in the second).
> 
> 
> While approach 3 works - what I would really like is to not have the
> camelLock placed on the files that are not being processed.
> 
> So watching the directory, there would be (at most) 5 files with camelLock
> files created at a time, when they finish they are moved to the .camel
> directory, and then it starts processing the next file in the directory.
> 
> Is that possible?  Is there anything I should be sure to do in an error
> route so that I "roll back" the camel locks to ensure that unprocessed
> files are ready to process the next time the application starts?
> 

Hi,

Maybe you can try to use the parameter maxMessagesPerPoll on the file
endpoint i.e.: 
from("file://incoming?maxMessagesPerPoll=5").thread(5).process() 

Check the file component documentation : http://camel.apache.org/file2.html

-- 
View this message in context: http://old.nabble.com/Processing-5-files-at-a-time---Threads--SEDA-%2B-Concurrent-Consumers--tp26960942p26965930.html
Sent from the Camel - Users mailing list archive at Nabble.com.


Mime
View raw message