hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13371) S3A globber to use bulk listObject call over recursive directory scan
Date Mon, 20 Mar 2017 17:21:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933107#comment-15933107

ASF GitHub Bot commented on HADOOP-13371:

Github user kazuyukitanimura commented on the issue:

    Thanks @steveloughran 
    I understand your point that getting the sign off for the core class changes is not easy.
At the same time, #204 seems to be a big change. I was wondering if there is a way to meet
at somewhere in the middle. It is important to provide the end users a way to glob things
on S3. It easily hits OOM with the current code.
    Meanwhile, I will keep trying to contribute to #204, which seems to be a right long term
    Also, I made a few other fixes related to S3A. My current employer just allowed me to
spend 20% of my time to contribute back to the community. I hope you don't mind that I mention
your name in the pull requests that I am going to file.

> S3A globber to use bulk listObject call over recursive directory scan
> ---------------------------------------------------------------------
>                 Key: HADOOP-13371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13371
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
> HADOOP-13208 produces O(1) listing of directory trees in {{FileSystem.listStatus}} calls,
but doesn't do anything for {{FileSystem.globStatus()}}, which uses a completely different
codepath, one which does a selective recursive scan by pattern matching as it goes down, filtering
out those patterns which don't match. Cost is O(matching-directories) + cost of examining
the files.
> It should be possible to do the glob status listing in S3A not through the filtered treewalk,
but through a list + filter operation. This would be an O(files) lookup *before any filtering
took place*.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message