hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4565) MultiFileInputSplit can use data locality information to create splits
Date Tue, 20 Jan 2009 15:27:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665438#action_12665438

Enis Soztutar commented on HADOOP-4565:

My 2 cents : 
CombineFileRecordReader should have a javadoc explaining the basic use case. 
changes in MultiFileSplit seems fine.
it would be better to use the o.a.h.mapred.lib package for this InputFormat. 
CombinedFileInputFormat makes MultiFileInputFormat obsolete, with little differences. We should
deprecate MFIF and MFIS. 

bq. if we can preserve node locality - then there is never any reason to use the base inputformats.
we can always use the combineinputformat and let it organize splits optimally. otherwise not
having node locality is something we are always going to struggle with.
In it's current form, FileInputFormat does not return splits larger than block size, unless
we set mapred.min.split.size. However when it does, it could benefit from node/rack locality.

> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>                 Key: HADOOP-4565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4565
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: CombineMultiFile.patch, CombineMultiFile2.patch, CombineMultiFile3.patch,
CombineMultiFile4.patch, CombineMultiFile5.patch, CombineMultiFile7.patch
> The MultiFileInputFormat takes a set of paths and creates splits based on file sizes.
Each splits contains a few files an each split are roughly equal in size. It would be efficient
if we can extend this InputFormat to create splits such each all the blocks in one split and
either node-local or rack-local.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message