hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5186) mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat to fail
Date Mon, 11 Nov 2013 19:41:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819284#comment-13819284
] 

Hudson commented on MAPREDUCE-5186:
-----------------------------------

SUCCESS: Integrated in Hadoop-trunk-Commit #4713 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4713/])
MAPREDUCE-5186. mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat
to fail. Contributed by Robert Parker and Jason Lowe (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1540813)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/JobSplitWriter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/split
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/split/TestJobSplitWriter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestBlockLimits.java


> mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat
to fail
> ----------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5186
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5186
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: job submission
>    Affects Versions: 2.0.4-alpha, 2.2.0
>            Reporter: Sangjin Lee
>            Assignee: Robert Parker
>            Priority: Critical
>             Fix For: 3.0.0, 2.3.0
>
>         Attachments: MAPREDUCE-5186v1.patch, MAPREDUCE-5186v2.patch, MAPREDUCE-5186v3.patch,
MAPREDUCE-5186v3.patch
>
>
> CombineFileInputFormat can easily create splits that can come from many different locations
(during the last pass of creating "global" splits). However, we observe that this often runs
afoul of the mapreduce.job.max.split.locations check that's done by JobSplitWriter.
> The default value for mapreduce.job.max.split.locations is 10, and with any decent size
cluster, CombineFileInputFormat creates splits that are well above this limit.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message