Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Message-ID: <10844157.348321294941945510.JavaMail.jira@thor>
Date: Thu, 13 Jan 2011 13:05:45 -0500 (EST)
From: "Bryan Duxbury (JIRA)" <jira@apache.org>
To: common-dev@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7103) When rack awareness script returns
 nothing, cluster stops working
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

When rack awareness script returns nothing, cluster stops working
-----------------------------------------------------------------

                 Key: HADOOP-7103
                 URL: https://issues.apache.org/jira/browse/HADOOP-7103
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Bryan Duxbury


This was an interesting one. Our rack awareness script contains a 1-1 mapping from host/ip to rack. We added a new rack's worth of machines without updating the awareness script, and when the script was called, it returned absolutely no results for the new machines.

This resulted in the surprising result that basically the entire cluster stopped working. Even tasks or blocks assigned to nodes with a valid rack seemed to fail. The errors were only detectable by looking in the namenode and jobtracker logs, making it take a while before we could figure out the problem. After fixing the rack awareness script, everything returned to normal operation.

It seems to me that either the error should be raised more aggressively, or a "default" rack should be assumed. This would keep simple mistakes from making the entire cluster unusable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.