hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-2216) speculation should normalize progress rates based on amount of input data
Date Thu, 09 Dec 2010 20:10:01 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joydeep Sen Sarma updated MAPREDUCE-2216:
-----------------------------------------

    Component/s: jobtracker

hard to believe there's not a jira open for this already - please close/redirect if that is
so.

> speculation should normalize progress rates based on amount of input data
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2216
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2216
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>            Reporter: Joydeep Sen Sarma
>
> We frequently see skews in data distribution both on the mappers and reducers. The small
ones finish quickly and the longer ones immediately get speculated. We should normalize progress
rates used by speculation with some metric correlated to the amount of data processed by the
task (like bytes read of rows processed). That will prevent these unnecessary speculations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message