hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-964) Inaccurate values in jobSummary logs
Date Fri, 25 Sep 2009 07:22:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759417#action_12759417
] 

Hemanth Yamijala commented on MAPREDUCE-964:
--------------------------------------------

bq. Also of course, we must always set a start time before starting a task...

This happens.

bq. Sounds like things are very messy. Can we refactor the code to make this much simpler
/ obvious and make it more likely to be correctly maintained?

Yes, things are messy. I would not attempt refactor in this patch. Times not being set is
not the only thing that's messy. The whole status update logic needs a relook. There are duplicate
data structures and states and phases meaning different things in different contexts floating
all around.

bq. I'd call no end time a serious ERROR. Not an INFO level issue. I'd be tempted to assert...
Although this might not be best to do while trying to stabilize the code. Maybe in the mainline
though...

+1.

> Inaccurate values in jobSummary logs
> ------------------------------------
>
>                 Key: MAPREDUCE-964
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-964
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Rajiv Chittajallu
>            Assignee: Sreekanth Ramakrishnan
>            Priority: Critical
>         Attachments: mapreduce-964-1.patch
>
>
> For some jobs the mapSlotSeconds is incorrect.
> negative value
> 09/09/01 18:31:44 INFOmapred.JobInProgress$JobSummary: jobId=job_200908270718_4568,submitTime=1251823543976,launchTime=1251823554310,finishTime=1251829904565,
           numMaps=7965,numSlotsPerMap=1,numReduces=40,numSlotsPerReduce=1,user=wile,queue=runner,status=SUCCEEDED,
        mapSlotSeconds=-2503133523,reduceSlotsSeconds=186536,clusterMapCapacity=11262,clusterReduceCapacity=3754
> or too high
> 09/09/02 23:59:57 INFO mapred.JobInProgress$JobSummary: jobId=job_200908270718_5861,submitTime=1251935672924,launchTime=1251935687698,finishTime=1251935997949,
           numMaps=1026,numSlotsPerMap=1,numReduces=10,numSlotsPerReduce=1,user=dfsload,queue=gridops,status=SUCCEEDED,
        
> mapSlotSeconds=1251949742,reduceSlotsSeconds=537,clusterMapCapacity=11262,clusterReduceCapacity=3754

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message