apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2274) AbstractFileInputOperator gets killed when there are a large number of files.
Date Thu, 10 Nov 2016 19:44:58 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654940#comment-15654940

ASF GitHub Bot commented on APEXMALHAR-2274:

GitHub user mattqzhang opened a pull request:


    APEXMALHAR-2274 Handle large number of files for AbstractFileInputOpe…

    @PramodSSImmaneni Please review

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mattqzhang/apex-malhar APEXMALHAR-2274

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #490
commit cb37761b62761d0e74f1c9b8c7c47491735e5421
Author: Matt Zhang <mattqzhang@gmail.com>
Date:   2016-11-10T19:38:51Z

    APEXMALHAR-2274 Handle large number of files for AbstractFileInputOperator


> AbstractFileInputOperator gets killed when there are a large number of files.
> -----------------------------------------------------------------------------
>                 Key: APEXMALHAR-2274
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2274
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: Munagala V. Ramanath
>            Assignee: Matt Zhang
> When there are a large number of files in the monitored directory, the call to DirectoryScanner.scan()
can take a long time since it calls FileSystem.listStatus() which returns the entire list.
Meanwhile, the AppMaster deems this operator hung and restarts it which again results in the
same problem.
> It should use FileSystem.listStatusIterator() [in Hadoop 2.7.X] or FileSystem.listFiles()
[in 2.6.X] or other similar calls that return
> a remote iterator to limit the number files processed in a single call.

This message was sent by Atlassian JIRA

View raw message