Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <7606365.1171418706382.JavaMail.jira@brutus>
Date: Tue, 13 Feb 2007 18:05:06 -0800 (PST)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-1018) Single lost heartbeat leads to a
 "Lost task tracker"
In-Reply-To: <21121287.1171394705511.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472961 ] 

Arun C Murthy commented on HADOOP-1018:
---------------------------------------

JobTracker does exactly what your suggested: acknowledge the duplicate status-update without processsing; it resends the 'prevHeartbeatRespone'  i.e. the lost heartbeat response to the TT. Maybe the log message there is misleading?

> Single lost heartbeat leads to a "Lost task tracker"
> ----------------------------------------------------
>
>                 Key: HADOOP-1018
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1018
>             Project: Hadoop
>          Issue Type: Bug
>    Affects Versions: 0.10.0, 0.11.2, 0.12.0
>         Environment: Nutch trunk/ (Hadoop 0.10.0), Linux, JDK 1.5, a cluster of 9 machines.
>            Reporter: Andrzej Bialecki 
>
> Under heavy load, task tracker may lose the heartbeat response from the JobTracker. Task tracker tries to resend the last heartbeat message, which job tracker treats as "duplicate" response and ignores. Since task tracker tries to resend the same heartbeat message, with the same id, over and over again, no "valid" messages reach the job tracker, so after a while it considers the task tracker to be lost. Task tracker cannot recover from this state and needs to be restarted.
> Looking at Hadoop trunk/ I believe this problem still may occur - in JobTracker.java.heartbeat():992 JobTracker should not ignore duplicate messages but acknowledge them without processing. This would cause the task tracker to sync back it's last heartbeat id with the last hearbeat id remembered in the job tracker.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.