hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhenhua Guo <jen...@gmail.com>
Subject setting "mapred.task.cache.levels" to 0 makes Hadoop stall
Date Mon, 20 Dec 2010 23:27:54 GMT
It seems that mapred.task.cache.levels is used by JobTracker to create
task caches for nodes at various levels. This makes data-locality
scheduling possible.
If I set mapred.task.cache.levels to 0 and use default network
topology, then mapreduce job will stall forever. The reason is
JobInProgress::findNewMapTask always returns -1. Field
"nonRunningMapCache" is empty and field "nonLocalMaps" is also empty.
I wonder whether it is designed to behave like that. Or when
mapred.task.cache.levels is set 0, Hadoop should fall back to some
default caching strategy. E.g. put all tasks into
JobInProgress::nonLocalMaps.

Thanks
Gerald

Mime
View raw message