hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-442) slaves file should include an 'exclude' section, to prevent "bad" datanodes and tasktrackers from disrupting a cluster
Date Thu, 15 Feb 2007 20:07:06 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473491
] 

dhruba borthakur commented on HADOOP-442:
-----------------------------------------

0. Nice work on removing the STOPPED state. Makes the system less complex.

1. In TestDecommission.waitNodeState, we wait for half second per iteration.
   If this setting causes many prints of the form "Waiting for node to change.."
   then maybe we can change the wait period to 1 second instead of half second.

2. In TestDecommission, we have the following logic:
      a. decommissionNode()
      b. waitNodeState(DECOMMISSION_INPROGRESS)
      c. commissionNode()
      d. waitNodeState(NORMAL)
      e. decommissionNode()
      f. waitNodeState(DECOMMISSIONED)
      g. checkFile()

  I guess b. should be  waitNodeState(DECOMMISSIONED)
  Also, we can sneak in a call to checkFile() betwen steps b and c.

3. Maybe ClientProtocol.refreshNodes can return a void instead of
   "boolean".

4. The original way to shutdown a datanode was to make the namenode send a
   DNA_SHUTDOWN command in response to a heartbeat. You enahnced this logic
   to make the datanode catch a UnregisteredDatanodeException and shut itself
   down. Thus, we will now have two ways to shutdown a datanode. Do you think
   that it is preferable to have only one way to shutdown a datanode?

5. An earlier patch introduced an async thread called the ReplicationMonitor.
   The ReplicationMonitor thread invokes checkDecommissionState(). This
   probably means that the DecommissionedMonitor thread is not needed anymore.

6. The FSNamesystem.verifyNodeRegistration needs to be synchronized since it
   traverses hosts lists and datanode lists. Since
   FSNamesystem.startDecommission and FSNamesystem.stopDecommission are private
   and are always called from already-synchronized methods, we can remove
   the "synchronized" keyword from their definitions.

7. I am kind-of reluctant to put in added complexity to pendingTransfers
   to handle the case that a decommissioned node should not be the
   source of a replication request. Especially because the above rule is
   not enforced if there is only one replica in the cluster. Is there any
   way that we can avoid putting in this special purpose case? What bad
   thing can occur if we select the decommissioned node as the source of
   a replication request?

8. Ideally, the FSNamesystem.verifyNodeShutdown needs to be synchronized
   because it looks up the datanode descriptor. But this method is called
   for every RPC request and a "synchronized" call just adds overhead.
   Maybe we can make FSNamesystem.gotHeartBeat, FSNamesystem.processReport
   and FSNamesystem.blockReceived call FSNamesystem.verifyNodeShutdown.


> slaves file should include an 'exclude' section, to prevent "bad" datanodes and tasktrackers
from disrupting  a cluster
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-442
>                 URL: https://issues.apache.org/jira/browse/HADOOP-442
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>            Reporter: Yoram Arnon
>         Assigned To: Wendy Chien
>         Attachments: hadoop-442-10.patch, hadoop-442-8.patch
>
>
> I recently had a few nodes go bad, such that they were inaccessible to ssh, but were
still running their java processes.
> tasks that executed on them were failing, causing jobs to fail.
> I couldn't stop the java processes, because of the ssh issue, so I was helpless until
I could actually power down these nodes.
> restarting the cluster doesn't help, even when removing the bad nodes from the slaves
file - they just reconnect and are accepted.
> while we plan to avoid tasks from launching on the same nodes over and over, what I'd
like is to be able to prevent rogue processes from connecting to the masters.
> Ideally, the slaves file will contain an 'exclude' section, which will list nodes that
shouldn't be accessed, and should be ignored if they try to connect. That would also help
in configuring the slaves file for a large cluster - I'd list the full range of machines in
the cluster, then list the ones that are down in the 'exclude' section

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message