hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-5186) mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat to fail
Date Sat, 27 Apr 2013 00:28:19 GMT
Sangjin Lee created MAPREDUCE-5186:
--------------------------------------

             Summary: mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat
to fail
                 Key: MAPREDUCE-5186
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5186
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv1, mrv2
    Affects Versions: 2.0.4-alpha
            Reporter: Sangjin Lee


CombineFileInputFormat can easily create splits that can come from many different locations
(during the last pass of creating "global" splits). However, we observe that this often runs
afoul of the mapreduce.job.max.split.locations check that's done by JobSplitWriter.

The default value for mapreduce.job.max.split.locations is 10, and with any decent size cluster,
CombineFileInputFormat creates splits that are well above this limit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message