hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4565) MultiFileInputSplit can use data locality information to create splits
Date Sat, 17 Jan 2009 01:47:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664777#action_12664777
] 

Zheng Shao commented on HADOOP-4565:
------------------------------------

Nitpick:

CombineFileRecordReader seems to contain an unnecessary recursive call, which can be easily
eliminated.

{code}
  public boolean next(K key, V value) throws IOException {
    while ((curReader == null) || !curReader.next(key, value)) {
      if (!initNextRecordReader()) {
        return false;
      }
    }
    return true;
  }
{code}


I am OK with that we only use rack locality. I think most statistics have shown that intra-rack
communication is nearly free, especially when we are reading big blocks of data.


> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4565
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: CombineMultiFile.patch, CombineMultiFile2.patch, CombineMultiFile3.patch,
CombineMultiFile4.patch, CombineMultiFile5.patch, CombineMultiFile7.patch
>
>
> The MultiFileInputFormat takes a set of paths and creates splits based on file sizes.
Each splits contains a few files an each split are roughly equal in size. It would be efficient
if we can extend this InputFormat to create splits such each all the blocks in one split and
either node-local or rack-local.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message