hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8234) DistributedFileSystem and Globber should apply PathFilter early
Date Tue, 26 May 2015 15:20:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559232#comment-14559232
] 

Hadoop QA commented on HDFS-8234:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  14m 38s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 1 new
or modified test files. |
| {color:green}+1{color} | javac |   7m 28s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |   9m 36s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   2m 12s | The applied patch generated  1 new checkstyle
issues (total was 4, now 4). |
| {color:red}-1{color} | checkstyle |   3m 21s | The applied patch generated  1 new checkstyle
issues (total was 48, now 48). |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that end in whitespace.
|
| {color:green}+1{color} | install |   1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with eclipse:eclipse.
|
| {color:green}+1{color} | findbugs |   4m 45s | The patch does not introduce any new Findbugs
(version 3.0.0) warnings. |
| {color:green}+1{color} | common tests |  23m  2s | Tests passed in hadoop-common. |
| {color:green}+1{color} | hdfs tests | 164m 11s | Tests passed in hadoop-hdfs. |
| | | 229m 32s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12735300/HDFS-8234.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9a3d617 |
| checkstyle |  https://builds.apache.org/job/PreCommit-HDFS-Build/11127/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-HDFS-Build/11127/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-common test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11127/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11127/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11127/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11127/console |


This message was automatically generated.

> DistributedFileSystem and Globber should apply PathFilter early
> ---------------------------------------------------------------
>
>                 Key: HDFS-8234
>                 URL: https://issues.apache.org/jira/browse/HDFS-8234
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>            Assignee: J.Andreina
>              Labels: newbie
>         Attachments: HDFS-8234.1.patch, HDFS-8234.2.patch
>
>
> HDFS-985 added partial listing in listStatus to avoid listing entries of large directory
in one go. If listStatus(Path p, PathFilter f) call is made, filter is applied after fetching
all the entries resulting in a big list being constructed on the client side. If the DistributedFileSystem.listStatusInternal()
applied the PathFilter it would be more efficient. So DistributedFileSystem should override
listStatus(Path f, PathFilter filter) and apply PathFilter early. 
> Globber.java also applies filter after calling listStatus.  It should call listStatus
with the PathFilter.
> {code}
> FileStatus[] children = listStatus(candidate.getPath());
>            .........
>             for (FileStatus child : children) {
>               // Set the child path based on the parent path.
>               child.setPath(new Path(candidate.getPath(),
>                       child.getPath().getName()));
>               if (globFilter.accept(child.getPath())) {
>                 newCandidates.add(child);
>               }
>             }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message