hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-270) TaskTracker could send an out-of-band heartbeat when the last running map/reduce completes
Date Fri, 25 Sep 2009 08:50:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759441#action_12759441
] 

Arun C Murthy commented on MAPREDUCE-270:
-----------------------------------------

Nigel - This patch proved very very hard to test without mock-objects. For now, I've attached
a slightly arbitrary test-case which checks does the following:

# Simulates a very large cluster by setting a very high value of 30s for the heartbeat-interval
between the JobTracker and TaskTracker.
# Switches on the out-of-band heartbeat for the cluster.
# Submits a very small random-writer job with 2 maps and asserts that the job completes within
120s. 

The 120s deadline is carefully chosen with the idea that a randomwriter job with 2 maps will
need at least 4 heartbeats: setup-task, map_0, map_1 and cleanup-task. However this is still
arbitrary and not very scientific. So, should we commit this test-case given that it is slightly
flaky? Thoughts?

PS: The job completes in ~50s with out-of-band heartbeats turned on, and in ~3mins with it
turned off. FYI

> TaskTracker could send an out-of-band heartbeat when the last running map/reduce completes
> ------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-270
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-270
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-270.patch, MAPREDUCE-270.patch, MAPREDUCE-270.patch, MAPREDUCE-270_yhadoop20.patch,
MAPREDUCE-270_yhadoop20.patch, MAPREDUCE-270_yhadoop20.patch
>
>
> Currently the TaskTracker strictly respects the heartbeat interval, this causes utilization
issues when all running tasks complete. We could send an out-of-band heartbeat in that case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message