hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-124) don't permit two datanodes to run from same dfs.data.dir
Date Wed, 10 May 2006 00:20:05 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-124?page=comments#action_12378787 ] 

Konstantin Shvachko commented on HADOOP-124:

I see exactly 2 problems now.
1) Two datanodes can be started sharing the same data directory, and reporting the same blocks
I'm planning to solve this by placing a lock in the data dir.
2) Starting a replacement datanode.
If a datanode fails, or intentionally shut down, the new datanode
server can be started on the same node and port or on a different node or port,
but connected to the same data directory.
In current implementation if the new datanode starts before the namenode removed the old datanode
its list identical block will be considered belonging to both nodes, and consequently over-replicated.
To solve this second problem the datanode should store a unique id per data directory as proposed.
And the namenode should identify block reports by those ids (only!), rather than by hostname:port.
Each datanode reporting the same id (no matter which hostname or port it has) should be considered
a replacement to the old node, and the old datanode at this point should be removed from the
the same way it is removed when its hearbeats are not reaching the namenode for 10 minutes.

> don't permit two datanodes to run from same dfs.data.dir
> --------------------------------------------------------
>          Key: HADOOP-124
>          URL: http://issues.apache.org/jira/browse/HADOOP-124
>      Project: Hadoop
>         Type: Bug

>   Components: dfs
>     Versions: 0.2
>  Environment: ~30 node cluster
>     Reporter: Bryan Pendleton
>     Assignee: Konstantin Shvachko
>     Priority: Critical
>      Fix For: 0.3

> DFS files are still rotting.
> I suspect that there's a problem with block accounting/detecting identical hosts in the
namenode. I have 30 physical nodes, with various numbers of local disks, meaning that my current
'bin/hadoop dfs -report" shows 80 nodes after a full restart. However, when I discovered the
 problem (which resulted in losing about 500gb worth of temporary data because of missing
blocks in some of the larger chunks) -report showed 96 nodes. I suspect somehow there were
extra datanodes running against the same paths, and that the namenode was counting those as
replicated instances, which then showed up over-replicated, and one of them was told to delete
its local block, leading to the block actually getting lost.
> I will debug it more the next time the situation arises. This is at least the 5th time
I've had a large amount of file data "rot" in DFS since January.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message