hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sreekanth Ramakrishnan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-964) Inaccurate values in jobSummary logs
Date Tue, 22 Sep 2009 08:26:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758171#action_12758171
] 

Sreekanth Ramakrishnan commented on MAPREDUCE-964:
--------------------------------------------------

The reason for the very high map slot seconds is because of following senario,

We hand out of a task to tracker, tracker gets lost, then we kill the task, as TT has not
reported back during lost period our task start time is not updated according to JT the task
has not been launched and the time and then we fail the task and the task has a finish time
but not start time.

Following is log from JT:

{noformat}
2009-09-18 13:28:20,990 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_200909180756_0009_m_000003_2'
to tip task_200909180756_0009_m_000003, for tracker 'tracker_xxx:localhost/127.0.0.1:45507'
2009-09-18 13:39:51,482 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_200909180756_0009_m_000003_2:
Lost task tracker: tracker_xxx:localhost/127.0.0.1:45507
2009-09-18 13:39:56,997 INFO org.apache.hadoop.mapred.JobTracker: Adding task (cleanup)'attempt_200909180756_0009_m_000003_2'
to tip task_200909180756_0009_m_000003, for tracker 'tracker_xxx:localhost/127.0.0.1:60187'
2009-09-18 13:40:00,000 INFO org.apache.hadoop.mapred.JobInProgress: TaskDebug attemptId :
attempt_200909180756_0009_m_000003_2 slots : SLOTS_MILLIS_MAPS tip.numslots is: 1 difference
to add : 1253281197003 status start : 0 status end time : 1253281197003
2009-09-18 13:40:00,000 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_200909180756_0009_m_000003_2'
from 'tracker_xxx:localhost/127.0.0.1:60187'
{noformat}

> Inaccurate values in jobSummary logs
> ------------------------------------
>
>                 Key: MAPREDUCE-964
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-964
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Rajiv Chittajallu
>            Assignee: Sreekanth Ramakrishnan
>            Priority: Critical
>
> For some jobs the mapSlotSeconds is incorrect.
> negative value
> 09/09/01 18:31:44 INFOmapred.JobInProgress$JobSummary: jobId=job_200908270718_4568,submitTime=1251823543976,launchTime=1251823554310,finishTime=1251829904565,
           numMaps=7965,numSlotsPerMap=1,numReduces=40,numSlotsPerReduce=1,user=wile,queue=runner,status=SUCCEEDED,
        mapSlotSeconds=-2503133523,reduceSlotsSeconds=186536,clusterMapCapacity=11262,clusterReduceCapacity=3754
> or too high
> 09/09/02 23:59:57 INFO mapred.JobInProgress$JobSummary: jobId=job_200908270718_5861,submitTime=1251935672924,launchTime=1251935687698,finishTime=1251935997949,
           numMaps=1026,numSlotsPerMap=1,numReduces=10,numSlotsPerReduce=1,user=dfsload,queue=gridops,status=SUCCEEDED,
        
> mapSlotSeconds=1251949742,reduceSlotsSeconds=537,clusterMapCapacity=11262,clusterReduceCapacity=3754

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message