hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
Date Thu, 19 Jun 2014 14:01:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037346#comment-14037346

Daryn Sharp commented on HDFS-5546:

I took a look at the source.  Ls doesn't stop during a descent.  It appears to only prematurely
abort when invoked with multiple args and one of those args disappears after the command starts.
 Ex.  "hadoop fs -ls /dir1 ...".  If /dir1 disappears, then it aborts w/o processing subsequent
paths which I agree is wrong.  Displaying FNF and returning non-zero is the correct behavior
though since the directory was explicitly requested to be listed.

The first patch appears to be the correct solution, but instead of catching only FNF and displaying
the error, I think it should catch IOE.

> race condition crashes "hadoop ls -R" when directories are moved/removed
> ------------------------------------------------------------------------
>                 Key: HDFS-5546
>                 URL: https://issues.apache.org/jira/browse/HDFS-5546
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>             Fix For: 3.0.0
>         Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch, HDFS-5546.2.001.patch,
> This seems to be a rare race condition where we have a sequence of events like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which calls DFS#listStatus(D).
This throws FileNotFoundException.
> 4. ls command terminates with FNF

This message was sent by Atlassian JIRA

View raw message