hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhenhua Guo <jen...@gmail.com>
Subject setting "mapred.task.cache.levels" to 0 makes Hadoop stall
Date Mon, 20 Dec 2010 23:27:54 GMT
It seems that mapred.task.cache.levels is used by JobTracker to create
task caches for nodes at various levels. This makes data-locality
scheduling possible.
If I set mapred.task.cache.levels to 0 and use default network
topology, then mapreduce job will stall forever. The reason is
JobInProgress::findNewMapTask always returns -1. Field
"nonRunningMapCache" is empty and field "nonLocalMaps" is also empty.
I wonder whether it is designed to behave like that. Or when
mapred.task.cache.levels is set 0, Hadoop should fall back to some
default caching strategy. E.g. put all tasks into


View raw message