apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Weise (JIRA)" <j...@apache.org>
Subject [jira] [Moved] (APEXMALHAR-2021) Add property to AbstractFileInputOperator to trim processedFiles and ignoredFiles
Date Fri, 18 Mar 2016 17:55:33 GMT

     [ https://issues.apache.org/jira/browse/APEXMALHAR-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Weise moved APEXCORE-343 to APEXMALHAR-2021:
---------------------------------------------------

    Affects Version/s:     (was: 3.3.0)
                       3.3.0
             Workflow: Default workflow, editable Closed status  (was: jira)
                  Key: APEXMALHAR-2021  (was: APEXCORE-343)
              Project: Apache Apex Malhar  (was: Apache Apex Core)

> Add property to AbstractFileInputOperator to trim processedFiles and ignoredFiles
> ---------------------------------------------------------------------------------
>
>                 Key: APEXMALHAR-2021
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2021
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>    Affects Versions: 3.3.0
>            Reporter: Munagala V. Ramanath
>            Assignee: Munagala V. Ramanath
>
> In AbstractFileInputOperator, the processedFiles and DirectoryScanner.ignoredFiles sets
can continue to grow without bound as new files are added to the monitored directory. There
are scenarios where an input file is deleted by the application once it is processed; here,
it is useful to provide a property that, if true, will cause the scanner to remove deleted
files from these sets to prevent unbounded growth.
> The only way to do this currently is to extend DirectoryScanner and provide implementations
of the scan() and createPartition() methods which forces the operator writer to unnecessarily
grapple with the internals of the base class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message