hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-1903) Allow different slowTaskThreshold for mappers and reducers
Date Thu, 01 Jul 2010 01:20:50 GMT
Allow different slowTaskThreshold for mappers and reducers
----------------------------------------------------------

                 Key: MAPREDUCE-1903
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1903
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: jobtracker
    Affects Versions: 0.22.0
            Reporter: Scott Chen
            Assignee: Scott Chen
             Fix For: 0.22.0


We have been running the new speculative logic in HADOOP-2141 done by Andy in our production
cluster.
One thing that we observed is that there are significantly low number of speculative reducers.
We have seen usually 100 speculative mappers launched per minute but speculative reducers
are usually less than 5.
But reducers are usually where we get complains about having speculative issues.

These two types of tasks has different properties and different needs.
It would be nice if we can configure the slow threshold separately to deal with them separately.

We can add the following config keys to allow setting them independently.
MRJobConfig.SPECULATIVE_MAP_SLOWTASK_THRESHOLD
MRJobConfig.SPECULATIVE_REDUCE_SLOWTASK_THRESHOLD

Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message