hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4565) MultiFileInputSplit can use data locality information to create splits
Date Mon, 12 Jan 2009 16:54:10 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662999#action_12662999
] 

Joydeep Sen Sarma commented on HADOOP-4565:
-------------------------------------------

Comments:
- min/maxSplitSize - i found the initialization confusing. is it the intent that values supplied
by setXXXSplitSize take precedence over config values? (I would find that logical - but the
implementation seems different)
- OneFileInfo - it seems to me that this class has no use except that it's constructor has
a side-effect of populating some external structures. As a style issue - this seems unnecessary
- we could just have a simple helper function instead.
- for(boolean doIterate ..): i found this variable pretty confusing. could we just replace
doIterator with blockToNodes.size()>0 ?

- one of the comments from the last review was that CombineFileSplit would be nice to treat
as an extension of MultiFileSplit. I would still make that argument - there should be a single
base class/interface for these multisplit classes - because then the CombineFileRecordReader
can be made to work with all of them. 

  Looking at the apis - it actually seems that MultiFileSplit can perhaps be treated as a
special implementation (extension) of CombineFileSplit (with get*offset functions always returning
0). 

  Any thoughts on this?

Not that familiar with miniMR stuff - would be good if someone else can review the test code
as well ..

> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4565
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: CombineMultiFile.patch, CombineMultiFile2.patch, CombineMultiFile3.patch,
CombineMultiFile4.patch, CombineMultiFile5.patch
>
>
> The MultiFileInputFormat takes a set of paths and creates splits based on file sizes.
Each splits contains a few files an each split are roughly equal in size. It would be efficient
if we can extend this InputFormat to create splits such each all the blocks in one split and
either node-local or rack-local.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message