hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
Date Sat, 18 Mar 2017 17:54:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931307#comment-15931307

ASF GitHub Bot commented on HADOOP-13371:

Github user steveloughran commented on the issue:

    OK, I see what you've done here: hidden a filter in Path and then used it later on. I
like the trick
    At the same time, I think it's not going to get past anyone else: its making a fundamental
change to a core class, one that get serialized around and created a lot. It's not going to
be allowed.
    That's OK though, for the following reason: we have the freedom to add/extend the methods
in S3aFS itself, so can do one which takes a filter as a parameter. If we do it right, we
can get this into the FS spec, or at least start negotiating on that topic (needs: spec, tests,
etc), while implementing it in S3AFS without waiting.

> S3A globber to use bulk listObject call over recursive directory scan
> ---------------------------------------------------------------------
>                 Key: HADOOP-13371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13371
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
> HADOOP-13208 produces O(1) listing of directory trees in {{FileSystem.listStatus}} calls,
but doesn't do anything for {{FileSystem.globStatus()}}, which uses a completely different
codepath, one which does a selective recursive scan by pattern matching as it goes down, filtering
out those patterns which don't match. Cost is O(matching-directories) + cost of examining
the files.
> It should be possible to do the glob status listing in S3A not through the filtered treewalk,
but through a list + filter operation. This would be an O(files) lookup *before any filtering
took place*.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message