hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-681) Adminstrative hook to pull live nodes out of a HDFS cluster
Date Mon, 06 Nov 2006 05:12:40 GMT
Adminstrative hook to pull live nodes out of a HDFS cluster

                 Key: HADOOP-681
                 URL: http://issues.apache.org/jira/browse/HADOOP-681
             Project: Hadoop
          Issue Type: New Feature
          Components: dfs
    Affects Versions: 0.8.0
            Reporter: dhruba borthakur
         Assigned To: dhruba borthakur

An administrator sometimes needs to bring down a datanode for scheduled maintenance. It would
be nice if HDFS can be informed about this event. On receipt of this event, HDFS can take
steps so that HDFS data is not lost when the node goes down at a later time.

In the existing architecture, a datanode can be in one of two states: dead or alive. A datanode
is alive if its heartbeats are being processed by the namenode. Otherwise that datanode is
in dead state. We extend the architecture to introduce the concept of a tranquil state for
a datanode.
A datanode is in tranquil state if:
    - it cannot be a target for replicating any blocks
    - any block replica that it currently contains does not count towards the target-replication-factor
of that block

Thus, a node that is in tranquil state can be brought down without impacting the guarantees
provided by HDFS.

The tranquil state is not persisted across namenode restarts. If the namenode restarts then
that datanode will go back to being in the dead or alive state.

The datanode is completely transparent to the fact that it has been labeled as being in tranquil
state. It can continue to heartbeat and serve read requests for datablocks.

DFSShell Design
We extend the DFS Shell utility to specify a list of nodes to the namenode.
    hadoop dfs -tranquil {set|clear|get} datanodename1 [,datanodename2]

The DFSShell utility sends this list to the namenode. This DFSShell command invoked with the
"set" option completes when the list is transferred to the namenode. This command is non-blocking;
it returns before the datanode is actually in the tranquil state. The client can then query
the state by re-issuing the command with the "get" option. This option will indicate whether
the datanode is in tranquil state or is "being tranquiled". The "clear" option is used to
transition a tranquil datanode to the alive state. The "clear" option is a no-op if the datanode
is not in the "tranquil" state.

ClientProtocol Design
The ClientProtocol is the protocol exported by the namenode for its client.
This protocol is extended to incorporate three new methods:
   ClientProtocol.setTranquil(String[] datanodes)
   ClientProtocol.getTranquil(String datanode)
   ClientProtocol.clearTranquil(String[] datanodes)

The ProtocolVersion is incremented to prevent conversations between imcompatible clients and
servers. An old DFSShell cannot talk to the new NameNode and vice-versa.

NameNode Design
The namenode does the bulk of the work for supporting this new feature.

The DatanodeInfo object has a new private member named "state". It also has three new member
    datanodeInfo.tranquilStarted(): start the process of tranquilization
    datanodeInfo.tranquilCompleted(): node is not in tranquil state
    datanodeInfo.clearTranquil() : remove tranquilization from node

The namenode exposes a new API to set and clear tranquil states for a datanode. On receipt
of a "set tranquil" command, it invokes datanodeInfo.tranquilStarted().

The FSNamesystem.chooseTarget() method skips over datanodes that are marked as being in the
"tranquil" state. This ensures that tranquil-datanodes are never chosen as targets of replication.
The namenode does *not* record
this operation in either the FsImage or the EditLogs.

The namenode puts all the blocks from a being-tranquiled node into the neededReplication data
structure. Necessary code changes are made to ensure that these blocks get replicated by the
regular replication method. As of now, the regular replication code does not distinguish between
these blocks and the blocks that are replication candidates because some other datanode might
have died. It might be prudent to give different (lower?) weightage to this type of replication
requests, but that exercise is deferred to a later date. In this design, replication requests
generated because of a node going to a tranquil state are not distinguished from replication
requests generated by a datanode going to the dead state.

The DatanodeInfo object has another new private member named "pendingTranquilCount". This
field stores the remaining number of blocks that still remain to be replicated. This field
is valid only if the node is in the ets being-tranquiled state.  On receipt of every 'n' heartbeats
from the being-tranquiled datanode, the namenode calculates the amount of data that is still
remaining to be replicated and updates the "pendingTranquilCount". in the DatanodeInfo.When
all the replications complete, the datanode is marked as tranquiled. The number 'n' is selected
in such a way that the average heartbeat processing time does not increase appreciably.

It is possible that the namenode might stop receving heartbeats from a datanode that is being-tranquiled.
In this case,   the tranquil flag of the datanode gets cleared. It transitions to the dead
state and the normal processing for alive-to-dead transition occurs here.

Web Interface
The dfshealth.jsp displays the live nodes, dead nodes, being-tranquiled and tranquil nodes.
For nodes in the being-tranquiled state, it displays the percentage of tranquilization completed
till now.

1. If a request for tranquilization starts getting processed and there aren't enough space
available in DFS to complete the necessary replication, then that node might remain in the
being-tranquiled state for a long long time. This is not necessarily a bad thing but is there
a better option?

2. We have opted for not storing cluster configuration information in the persistent image
of the file system. (The tranquil state of a datanode may be lost if the namenode restarts).

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message