camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: File Consumer with dynamic list of files to poll
Date Mon, 22 Feb 2010 08:34:26 GMT
On Mon, Feb 22, 2010 at 9:04 AM, wmoussel <wmoussel@gmail.com> wrote:
>
> Hi,
>
> Here's my initial issue. I have to poll for new files in a directory
> containing a lot of files (5000 to 20000) (and I can't move them after
> processing).
>
> When using idempotent option it takes way too much cpu (like 9% even when
> they aren't new files).
>

How often are you gonna poll in those files? You can probably change
the delay to be less frequent.

In any case somehow you gotta match all those 20000 files whether or
not you have processed them before.


> So i tried adding a filter so that the consumer would only take files less
> than 2  days old. It doesn't change much in the end. When I looked into how
> filter works in Java I got why it's still cpu consuming...
>
> So then I thought i would get recent file names with a unix command like
> find . -mtime -1 and use pollenrich to poll specific files. The way I
> understand how poll enrich works, it will poll the whole directory and then
> try to match filename afterwards. That is also cpu consuming since it gets
> all the files as exchanges prior to the match. Am I wrong ?
>

pollEnrich is not suitable for this as its meant for polling and
aggregating 1 resource at a time.

> So I'm basically trying to invoke a FileName to GenericFileExchange
> processor in the middle of my route. Is there a way to do this?
>
> I'm open to any suggestion.
>

The file component supports using a custom PollingConsumerPollStrategy
http://camel.apache.org/polling-consumer.html

You can then implement your own logic and return false in the being
method where there are no new files since last time.


> Thanks in advance :)
>
> Wandrille
> --
> View this message in context: http://old.nabble.com/File-Consumer-with-dynamic-list-of-files-to-poll-tp27683938p27683938.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message