hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Roelofs <roel...@yahoo-inc.com>
Subject Re: setting "mapred.task.cache.levels" to 0 makes Hadoop stall
Date Thu, 06 Jan 2011 00:34:12 GMT
Zhenhua Guo <jenvor@gmail.com> wrote:

> It seems that mapred.task.cache.levels is used by JobTracker to create
> task caches for nodes at various levels. This makes data-locality
> scheduling possible.
> If I set mapred.task.cache.levels to 0 and use default network
> topology, then mapreduce job will stall forever. The reason is
> JobInProgress::findNewMapTask always returns -1. Field
> "nonRunningMapCache" is empty and field "nonLocalMaps" is also empty.
> I wonder whether it is designed to behave like that. Or when
> mapred.task.cache.levels is set 0, Hadoop should fall back to some
> default caching strategy. E.g. put all tasks into
> JobInProgress::nonLocalMaps.

I think there should either be a fallback mechanism or the code should
disallow/ignore values less than 1.  Can you file a JIRA issue for this?


View raw message