hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhenhua Guo <jen...@gmail.com>
Subject Re: setting "mapred.task.cache.levels" to 0 makes Hadoop stall
Date Fri, 01 Apr 2011 21:00:21 GMT
Sorry for replying so late.  I just created an JIRA ticket -
https://issues.apache.org/jira/browse/MAPREDUCE-2419.

Gerald

On Wed, Jan 5, 2011 at 7:34 PM, Greg Roelofs <roelofs@yahoo-inc.com> wrote:
> Zhenhua Guo <jenvor@gmail.com> wrote:
>
>> It seems that mapred.task.cache.levels is used by JobTracker to create
>> task caches for nodes at various levels. This makes data-locality
>> scheduling possible.
>> If I set mapred.task.cache.levels to 0 and use default network
>> topology, then mapreduce job will stall forever. The reason is
>> JobInProgress::findNewMapTask always returns -1. Field
>> "nonRunningMapCache" is empty and field "nonLocalMaps" is also empty.
>> I wonder whether it is designed to behave like that. Or when
>> mapred.task.cache.levels is set 0, Hadoop should fall back to some
>> default caching strategy. E.g. put all tasks into
>> JobInProgress::nonLocalMaps.
>
> I think there should either be a fallback mechanism or the code should
> disallow/ignore values less than 1.  Can you file a JIRA issue for this?
> https://issues.apache.org/jira/secure/CreateIssue!default.jspa
>
> Thanks,
>  Greg
>

Mime
View raw message