hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "philo vivero (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1960) dfs.*.dir should not default to /tmp (or other typically volatile storage)
Date Thu, 19 May 2011 01:34:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035916#comment-13035916
] 

philo vivero commented on HDFS-1960:
------------------------------------

I'm not 100% sure, but I believe dfs.name.dir data disappearing will cause the NameNode to
become entirely useless after a service restart. At least that is what appeared to happen
when I tried it.


> dfs.*.dir should not default to /tmp (or other typically volatile storage)
> --------------------------------------------------------------------------
>
>                 Key: HDFS-1960
>                 URL: https://issues.apache.org/jira/browse/HDFS-1960
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.20.2
>         Environment: *nix systems
>            Reporter: philo vivero
>            Priority: Critical
>
> The hdfs-site.xml file possibly will not define one or both of:
> dfs.name.dir
> dfs.data.dir
> If they are not specified, data is stored in /tmp. This is extremely dangerous. Rationale:
the cluster will work fine for days, possibly even weeks, before blocks will start to go missing.
Rebooting a datanode on common Linux systems will clear all the data from that node. There
is no documented way (that I'm aware of) to recover the situation. The cluster must be completely
obliterated and rebuilt from scratch.
> Better reactions to the missing configuration parameters:
> 1. DataNode dies on startup and asks that these parameters be defined.
> 2. Default is /var/db/hadoop (or some other non-volatile storage location). Naturally,
inability to write into that directory leads to DataNode to die on startup, logging error.
> The first solution would be most likely preferred by typical enterprise sysadmins. The
second solution is suboptimal (since /var/db/hadoop might not be the optimal location for
the data) but is still preferable to the current implementation, since it will less often
lead to an irretrievably corrupt cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message