hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerald Guo (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-2419) Jobs stall forever if mapred.task.cache.levels is set to 0.
Date Fri, 01 Apr 2011 20:59:05 GMT
Jobs stall forever if mapred.task.cache.levels is set to 0.

                 Key: MAPREDUCE-2419
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2419
             Project: Hadoop Map/Reduce
          Issue Type: Task
          Components: jobtracker
    Affects Versions: 0.21.0, 0.20.1
            Reporter: Gerald Guo
            Priority: Minor

It seems that mapred.task.cache.levels is used by JobTracker to create task caches for nodes
at various levels. This makes data-locality scheduling possible.
If I set mapred.task.cache.levels to 0 (to disable task caching) and use default network topology,
then mapreduce job will stall forever. The reason is JobInProgress::findNewMapTask always
returns -1. Field "nonRunningMapCache" is empty and field "nonLocalMaps" is also empty.  

I can think of two possible fixes. 
1) If mapred.task.cache.levels is set 0, Hadoop should fall back to some default caching strategy.
E.g. put all tasks into JobInProgress::nonLocalMaps.
2) ignore values less than 1

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message