hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
Date Fri, 03 Jun 2011 08:21:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043247#comment-13043247

Matt Foley commented on HADOOP-7327:

We have two instances of this bug actually causing NPEs in the real world (or at least the
Test world).

The first is in TestTrash.testTrashEmptier(), which runs on the LocalFileSystem, and launches
a Trash$Emptier thread in background.  Every time that thread cycles, it attempts to empty
the local .Trash directory in EVERY "home directory" known.  On my Mac, there are four user
accounts, each with its own "home" directory, one of which, /Users/test, has a .Trash directory
with access permissions set so it is not readable by me.  When it tries to list the .Trash
subdirectories with FileSystem.listStatus(), it logs the following:

11/05/23 14:34:54 WARN fs.Trash: RuntimeException during Trash.Emptier.run() java.lang.NullPointerException
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1114)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1137)
	at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494)
	at org.apache.hadoop.fs.Trash.expunge(Trash.java:215)
	at org.apache.hadoop.fs.Trash$Emptier.run(Trash.java:313)
	at java.lang.Thread.run(Thread.java:680)

The second example is from Daryn, and was partly described in HADOOP-7345.  When he tries
to use "bin/hadoop fs -ls aaa" on any directory "aaa" for which he does not have access privs,
it logs:

    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1115)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1150)
    at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494)
    at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:163)

It is seen that both of these cases go through the FileSystem.listStatus(ArrayList<FileStatus>,
Path, PathFilter) form of the call, and it is here that the attached patch is targeted.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access
permission failure
> ---------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
> Many processes that call listStatus() expect to handle IOException, but instead are getting
runtime error NullPointerException, if the directory being scanned is visible but no-access
to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar
is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message