Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <23895097.1170368225750.JavaMail.jira@brutus>
Date: Thu, 1 Feb 2007 14:17:05 -0800 (PST)
From: "Wendy Chien (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-442) slaves file should include an
 'exclude' section, to prevent "bad" datanodes and tasktrackers from
 disrupting  a cluster
In-Reply-To: <650567.1155195793928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469597 ] 

Wendy Chien commented on HADOOP-442:
------------------------------------

-I'll change the names to the ones Sameer proposed and make the default an empty string.

-The exclude/hosts files expect full hostnames listed, separated by whitespace (similar to the slaves file).  I can refine this if it's too cumbersome. 

-Some of us had a discussion about exclude and decommission.  The conclusion is that they should be combined.   

Below is the new design:

1. Namenode behavior in dealing with exclude/hosts lists.
  a. When the namenode starts up, it reads the hosts and exclude files.  When a node on the exclude list registers with the namenode, we mark it to be decommissioned.  When nodes are done being decommissioned, they are shutdown.  Nodes not on the include list will not be allowed to register.   
  b. When the namenode gets a refreshNodes command, it will update the hosts and exclude lists.  If a node is added to the hosts lists, then it will be allowed to register.  If a node is removed from the hosts list, then any further communication will be disallowed and it will be asked to shutdown.  If a node is added to the exclude list, then it will start to be decommissioned.  If a node is removed from the exlcude list, then the decommission process will be stopped.   

2. Decommissioning a node behaves slightly differently from before. 
  a. When a node is being decommissioned, we do not want to use its copies to replicate unless no copies exist on non-decommissioned nodes. 
  b. A new thread will be used to periodically check if a node is done being decommissioned.  If it is, the next time the node heartbeats, it will be told to shut down. 

Comments welcome (and please let me know if I forgot anything).  

> slaves file should include an 'exclude' section, to prevent "bad" datanodes and tasktrackers from disrupting  a cluster
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-442
>                 URL: https://issues.apache.org/jira/browse/HADOOP-442
>             Project: Hadoop
>          Issue Type: Bug
>          Components: conf
>            Reporter: Yoram Arnon
>         Assigned To: Wendy Chien
>
> I recently had a few nodes go bad, such that they were inaccessible to ssh, but were still running their java processes.
> tasks that executed on them were failing, causing jobs to fail.
> I couldn't stop the java processes, because of the ssh issue, so I was helpless until I could actually power down these nodes.
> restarting the cluster doesn't help, even when removing the bad nodes from the slaves file - they just reconnect and are accepted.
> while we plan to avoid tasks from launching on the same nodes over and over, what I'd like is to be able to prevent rogue processes from connecting to the masters.
> Ideally, the slaves file will contain an 'exclude' section, which will list nodes that shouldn't be accessed, and should be ignored if they try to connect. That would also help in configuring the slaves file for a large cluster - I'd list the full range of machines in the cluster, then list the ones that are down in the 'exclude' section

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.