hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramkumar Vadali (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-2185) Infinite loop at creating splits using CombineFileInputFormat
Date Mon, 23 May 2011 20:05:47 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ramkumar Vadali updated MAPREDUCE-2185:
---------------------------------------

    Attachment: MAPREDUCE-2185.patch

For blocks that do not have hosts associated with them, use NetworkTopology.DEFAULT_RACK as
the rack location. This avoids the infinite loop later on in getMoreSplits()

> Infinite loop at creating splits using CombineFileInputFormat
> -------------------------------------------------------------
>
>                 Key: MAPREDUCE-2185
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: job submission
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: MAPREDUCE-2185.patch
>
>
> This is caused by a missing block in HDFS. So the block's locations are empty. The following
code adds the block to blockToNodes map but not to rackToBlocks map. Later on when generating
splits, only blocks in rackToBlocks are removed from blockToNodes map. So blockToNodes map
can never become empty therefore causing infinite loop
> {code}
>           // add this block to the block --> node locations map
>           blockToNodes.put(oneblock, oneblock.hosts);
>           // add this block to the rack --> block map
>           for (int j = 0; j < oneblock.racks.length; j++) {
>              ..
>           }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message