camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hartmut Lang <hartmut.l...@ericsson.com>
Subject RE: Howto use maxMessagesPerPoll and sortBy for file-consumer
Date Thu, 19 Jan 2012 11:07:44 GMT
Ok,

maybe it's a good idea to add this to the documentation for file2, "Sorting using sortBy"
section.  

Thanks,
Hartmut Lang

-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com] 
Sent: Donnerstag, 19. Januar 2012 12:02
To: users@camel.apache.org
Subject: Re: Howto use maxMessagesPerPoll and sortBy for file-consumer

On Thu, Jan 19, 2012 at 11:57 AM, Hartmut Lang <hartmut.lang@ericsson.com> wrote:
> Hi,
>
> what i see from https://issues.apache.org/jira/browse/CAMEL-2899 is that maxMessagesPerPoll
and sortBy option do not work well together for a FileConsumer endpoint.
>
> In our use-case we have several thousand files in a from("file:...") route. And every
minute we want to process one file sorted by name.
> So we use the options: 
> delay=60000&maxMessagesPerPoll=1&sortBy=file:name
>
> But the way it seems to be implemented is:
> - read one file from fileystem
> - sort this file by name (so nothing happens here)
> - process this file
>
> The sortBy does not have any effect here, right?
>

Yes it used to sort before limiting. But then we had people with 1000000's of files, which
took to long time to sort, as the sorting was in-memory. And the FTP component re-uses the
logic, and the ftp file entries eat up more memory, so that was even slower.

So the file component will limit first, then sort afterwards.


> Can someone propose another solution where we can have the files processed sorted by-name
and also control how fast they are processed?
>
> Thanks,
> Hartmut
>



--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Mime
View raw message