hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3710) last split generated by FileInputFormat.getSplits may not have the best locality
Date Wed, 25 Jan 2012 13:40:41 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193033#comment-13193033
] 

Hudson commented on MAPREDUCE-3710:
-----------------------------------

Integrated in Hadoop-Mapreduce-0.23-Build #171 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/171/])
    MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split.
Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1235510 ../../trunk/

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235511
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java

                
> last split generated by FileInputFormat.getSplits may not have the best locality
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3710
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv1, mrv2
>    Affects Versions: 0.23.0, 1.0.0
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>             Fix For: 0.23.1
>
>         Attachments: MR-3710_v1.txt
>
>
> The last split generated by FileInputFormat.getSplits considers {{blkLocations.length-1}}
to be the hosts for the split.
> The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in which case
locality is picked up from a smaller block.
> e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. The hosts
for locality end up being the nodes containing the 3MB block instead of the 128MB block.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message