camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-11656) Support default directory sorter for FileConsumer
Date Thu, 23 Nov 2017 07:55:00 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263928#comment-16263928
] 

ASF GitHub Bot commented on CAMEL-11656:
----------------------------------------

davsclaus commented on issue #2108: CAMEL-11656 - add preSort option
URL: https://github.com/apache/camel/pull/2108#issuecomment-346549840
 
 
   Maybe we can improved the documentation a bit, something like:
   ```
   When pre-sort is enabled then the consumer will sort the file and directory names during
polling, that was retrieved from the file system. You may want to do this in case you need
to operate on the files in a sorted order. The pre-sort is executed before the consumer starts
to filter, and accept files to process by Camel. This option is default disabled.
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Support default directory sorter for FileConsumer
> -------------------------------------------------
>
>                 Key: CAMEL-11656
>                 URL: https://issues.apache.org/jira/browse/CAMEL-11656
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-core
>    Affects Versions: 2.19.0
>            Reporter: Viktor Khoroshko
>            Priority: Minor
>             Fix For: 2.21.0
>
>
> Hello,
> The current way of sorting consumed files isn't very flexible due to how it's implemented:
> Files are sorted AFTER they're added to in progress repository what in combination with
*maxMessagesPerPoll* set to be > 0  and *eagerLimitMaxMessagesPerPoll* set to false forces
the *removeExcessiveInProgressFiles* call - which in case of persistent idempotent repository
used causes redundant calls to a database. 
> This is not only the issue.
> In my case I have a logic in a custom file filter that a file shouldn't be processed
if a file with a same prefix is already in progress while still those files should be processed
in a specified order. 
> The issue is that since sorting is performed after each file is added to in progress
repository and before added there a file should be accepted by a filter but it will be not
 - as a previous file with a same prefix has already been added to in progress repo.
> *Example*: files  - test.001, test.002, test.003
> If test.003 is first in a returned file list then it will be added to in progress repository
and next files will not be accepted due to the same prefix. 
> Default sorting would solve this issue. 
> It would be great if default sorter can be specified for *FileConsumer* in the pollDirectory
method:
> {code:java}
>   log.trace("Polling directory: {}", directory.getPath());
>         File[] dirFiles = directory.listFiles();
>         if (dirFiles == null || dirFiles.length == 0) {
>             // no files in this directory to poll
>             if (log.isTraceEnabled()) {
>                 log.trace("No files found in directory: {}", directory.getPath());
>             }
>             return true;
>         } else {
>             // we found some files
>             if (log.isTraceEnabled()) {
>                 log.trace("Found {} in directory: {}", dirFiles.length, directory.getPath());
>             }
>         }
>         List<File> files = Arrays.asList(dirFiles);
>        // sort there?
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message