hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinay Kumar Thota (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1794) Test the job status of lost task trackers before and after the timeout.
Date Wed, 19 May 2010 12:19:56 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869112#action_12869112

Vinay Kumar Thota commented on MAPREDUCE-1794:

 The JobStatus should be JobStatus.FAILED instead of succeeded. If the task tracker was lost
for all the four attempts of a task should'nt the job fail instead of succeed, if that is
not the
case the message in the assert has to be changed the job suceeded even when loosing task tracker
for 4 times. 
I think you misunderstood the functionality. If tasktracker was lost and it wait for timeout,
later that task was marked as a killed and resubmitting into another task tracker. Even if
it kills for four attempts due to lost tasktracker, it will resubmitting to another tasktracker
for 5th time and keep continues until task succeed. For Killed tasks mapred.map.max.attempts
attribute won't applicable,so it attempts the task 'N' no.of times. Max attempts is only applicable
for failed tasks. In this case the job status should be succeed because of task might succeed
at one point of time.

 Why do we care for checking the job status for 40 % completion, also can be enhance the
building blocks to check this kind of status, since the code can be reused elsewhere. 
[Vinay] : We just wanted to make sure, the job should start and completes atleast 40% because,
atleast one  map or reduce tasks should run on the  tasktracker for checking the conditions.

The above code is repeated coupld of times can be part of a function, if this is used accross
test cases then can be part of building block. 
[Vinay] : I will refactor the code by making the function.I don't thinks so it useful across
the testcases.

If you see the story description we said we will suspend the task tracker and resume it, but
it seems that you have followed the route of killing the task tracker instead of pausing and
resuming it.
I think kiling should be fine since kill/start it emaulates the pause and resume, but on the
performance side if we had used pause and resume, so the waits in the test cases can be
[Vinay] : I am pausing by stoping the tasktracker and resuming it by starting the tasktracker.So
I don't think there would be a performance issue.

One general question I have is after killing the same task tracker 4 times, the task tracker
get blacklisted, and if you resubmit the job again, the task tracker should not be used by
job tracker.
Is it good to check that condition as part of this test case or do you think this is out of
There is url which has the blacklisted tasktracker, if we can get the number through aspect
it can be verified. Also at the end of the test we need to remove the task tracker from blacklisted
condition for the other tests to run without any problem. 
[Vinay] : for killed tasks, max attempts won't applicable like I said above. So there won't
be any blacklisted.

> Test the job status of lost task trackers before and after the timeout.
> -----------------------------------------------------------------------
>                 Key: MAPREDUCE-1794
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1794
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1794_lost_tasktracker.patch
> This test covers the following scenarios.
> 1. Verify the job status whether it is succeeded or not when  the task tracker is lost
and alive before the timeout.
> 2. Verify the job status and killed attempts of a task whether it is succeeded or not
and killed attempts are matched or not  when the task trackers are lost and it timeout for
all the four attempts of a task. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message