Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1973731873.1227890744527.JavaMail.jira@brutus>
Date: Fri, 28 Nov 2008 08:45:44 -0800 (PST)
From: "Enis Soztutar (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4565) MultiFileInputSplit can use data
 locality information to create splits
In-Reply-To: <525654219.1225490984230.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651600#action_12651600 ] 

Enis Soztutar commented on HADOOP-4565:
---------------------------------------

I agree with Joydeep that if this inputFormat is intended for handling large number of files, then the changes should be done in the MultiFileInputFormat itself, rather than introducing a new class. However a quick glance to the patch indicates that this inputformat works for files having more than one block. 
I guess we should better introduce the file[] -> blocks[] -> {hosts[], racks[]} -> splits[] logic in FileInputFormat and use this logic both in FileInputFormat and MultiFileInputFormat. 

I have opened an issue for a concrete MultiFileInputFormat implementation (HADOOP-4741), and attached a patch which I developed some time ago. HADOOP-4741 depends on this issue. 

> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4565
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: CombineMultiFile.patch, CombineMultiFile2.patch, CombineMultiFile3.patch
>
>
> The MultiFileInputFormat takes a set of paths and creates splits based on file sizes. Each splits contains a few files an each split are roughly equal in size. It would be efficient if we can extend this InputFormat to create splits such each all the blocks in one split and either node-local or rack-local.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.