[ https://issues.apache.org/jira/browse/HDFS-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124466#comment-13124466 ] 

Aaron T. Myers commented on HDFS-2422:
--------------------------------------

Looks like this is happening because {{o.a.h.fs.DF}} will return 0 for "space available" on a directory which doesn't exist:

{noformat}
[01:29:11] atm@simon:~$ hadoop org.apache.hadoop.fs.DF /
df -k null
null	72718632	49480712	19543996	73%	null
[01:29:23] atm@simon:~$ hadoop org.apache.hadoop.fs.DF /foo/bar/baz
df -k null
null	0	0	0	0%	null
{noformat}

I'm guessing the particular {{dfs.name.dir}} the NN was writing to was in fact a subdirectory of the mount directory, so when the NFS mount went away so did the subdirectory, causing DF to return 0.

I think this is indicative of a more basic issue with the {{NNResourceChecker}} policy, though. When syncing edit logs, the NN is designed to tolerate failure of up to N-1 {{dfs.name.dirs}}, but the {{NNResourceChecker}} will put the NN into safemode if only a single {{dfs.name.dir}} is low on space. The appropriate solution, then, seems to me to be to change the {{NNResourceChecker}} to also tolerate up to N-1 directories being low on space.

I'll create a patch to do this and upload it shortly.
                
> temporary loss of NFS mount causes NN safe mode
> -----------------------------------------------
>
>                 Key: HDFS-2422
>                 URL: https://issues.apache.org/jira/browse/HDFS-2422
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.24.0
>            Reporter: Jeff Bean
>            Assignee: Aaron T. Myers
>
> We encountered a situation where the namenode dropped into safe mode after a temporary outage of an NFS mount.
> At 12:10 the NFS server goes offline
> Oct  8 12:10:05 <namenode> kernel: nfs: server <nfs host> not responding, timed out
> This caused the namenode to conclude resource issues:
> 2011-10-08 12:10:34,848 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume '<nfs host>' is 0, which is below the configured reserved amount 104857600
> Temporary loss of NFS mount shouldn't cause safemode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira