hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HADOOP-5271) JobConf option for minimum progress threshold before reducers are assigned
Date Tue, 25 May 2010 04:59:27 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arun C Murthy resolved HADOOP-5271.

    Resolution: Fixed

Fixed via HADOOP-3136.

> JobConf option for minimum progress threshold before reducers are assigned
> --------------------------------------------------------------------------
>                 Key: HADOOP-5271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5271
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Tim Williamson
>         Attachments: HADOOP-5271.patch
> A specific sub-case of the general priority inversion problem noted in HADOOP-4557 is
when many lower priority jobs are submitted and are waiting for mappers to free up.  Even
though they haven't actually done any work, they will be assigned any free reducers.  If a
higher priority job is submitted, priority inversion results not just due to the lower priority
tasks that are in the midst of completing, but also due to the ones that haven't yet started
but have claimed all the free reducers.  A simple workaround is to require a job to complete
some useful work before assigning it a reducer.  This can be done in a tunable and backwards
compatible manner by adding a "minimum map progress percentage before assigning a reducer"
option to the JobConf.  Setting this to 0 would eliminate the common case above, and setting
it to 100 would technically eliminate the inversion of HADOOP-4557, though likely at an unacceptably
high cost.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message