hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Zhuge (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7352) FileSystem#listStatus should throw IOE upon access error
Date Thu, 08 Sep 2016 04:52:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472754#comment-15472754

John Zhuge commented on HADOOP-7352:

Move review comments from HADOOP-13191 to here.

Steve commented:
bq. Be aware that the FS shell expects globStatus to return null in certain conditions: someone
needs to look at all uses of this call and make sure that it isn't being used

John commented:

Went through 99 usages of {{globStatus(Path)}} in the following components: hadoop-common,
hadoop-distcp, hadoop-rumen, hadoop-streaming, hadoop-yarn-*, hadoop-hdfs (test only).

Most {{globStatus}} callers check both {{listFileStatus not null}} and {{listFileStatus.length
> 0}}, or expect {{listFileStatus}} to be not {{null}}, except:
  public static PathData[] expandAsGlob(String pattern, Configuration conf)
  throws IOException {
    Path globPath = new Path(pattern);
    FileSystem fs = globPath.getFileSystem(conf);    
    FileStatus[] stats = fs.globStatus(globPath);
    PathData[] items = null;
    if (stats == null) {
      // remove any quoting in the glob pattern
      pattern = pattern.replaceAll("\\\\(.)", "$1");
      // not a glob & file not found, so add the path with a null stat
      items = new PathData[]{ new PathData(fs, pattern, null) };
    } else {

I will include the fix for {{expandAsGlob}} in the next patch.

> FileSystem#listStatus should throw IOE upon access error
> --------------------------------------------------------
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 2.6.0
>            Reporter: Matt Foley
>            Assignee: John Zhuge
>         Attachments: HADOOP-7352.001.patch
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException
instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return
null, when the target directory exists but does not grant read permission.  This causes NPE
in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and
its linked issues for examples.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message