hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Dahiya (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-506) job tracker hangs on to dead task trackers "forever"
Date Thu, 05 Oct 2006 19:02:24 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-506?page=comments#action_12440217 ] 
            
Sanjay Dahiya commented on HADOOP-506:
--------------------------------------

One case in which I am able to reproduce this repeatedly is that job tracker restarts when
tasktrackers are still running. basically UNKNOWN_TASKTRACKER status messages are not handled
properly in job tracker. 

here is what happens in that case - 

synchronized (taskTrackers) {
    synchronized (trackerExpiryQueue) {
        boolean seenBefore = updateTaskTrackerStatus(trackerName,
                                                     trackerStatus);
        if (initialContact) {		                                        <<<<====
This is false 
            // If it's first contact, then clear out any state hanging around
            if (seenBefore) {		
                lostTaskTracker(trackerName, trackerStatus.getHost());
            }
        } else {
            // If not first contact, there should be some record of the tracker
            if (!seenBefore) {
                return InterTrackerProtocol.UNKNOWN_TASKTRACKER;    
                                                  <<<<=== returns this, but TT
already in tasktrackers and not in expiryQueue
            }
        }

        if (initialContact) {
            trackerExpiryQueue.add(trackerStatus);	                         <<<<====
not called 
        }
    }
}


in updateTaskTrackerStatus if (oldStatus == null && initialContact == false ) then
its a rogue status and should not be added to tasktrackers map in job tracker. 

I am investigating if this can happen in some other condition as well.


> job tracker hangs on to dead task trackers "forever"
> ----------------------------------------------------
>
>                 Key: HADOOP-506
>                 URL: http://issues.apache.org/jira/browse/HADOOP-506
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Yoram Arnon
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>
> I see cases where a task tracker gets disconnected from the job tracker and disconnects,
and then appears twice in the job tracker's list, with one instance being alive and well,
and the other's 'time since last heartbeat' increasing monotonically.
> that all makes sense.
> What doesn't make sense, is that the old instances never expire. It's been over 400000
seoncds since the last heartbeat. And the cluster reports having more nodes up and running
than its size (350 nodes in a 320 node cluster).
> there should be some reasonable timout for these expired task trackers, somewhere between
10 minutes and an hour.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message