Infinite loop at creating splits using CombineFileInputFormat
-------------------------------------------------------------
Key: MAPREDUCE-2185
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.22.0
This is caused by a missing block in HDFS. So the block's locations are empty. The following
code adds the block to blockToNodes map but not to rackToBlocks map. Later on when generating
splits, only blocks in rackToBlocks are removed from blockToNodes map. So blockToNodes map
can never become empty therefore causing infinite loop
{code}
// add this block to the block --> node locations map
blockToNodes.put(oneblock, oneblock.hosts);
// add this block to the rack --> block map
for (int j = 0; j < oneblock.racks.length; j++) {
..
}
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
|