hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "william zhu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-12684) CombineFileInputFormat may cause splits uneveness
Date Thu, 31 Dec 2015 01:59:49 GMT
william zhu created HADOOP-12684:

             Summary: CombineFileInputFormat may cause splits uneveness
                 Key: HADOOP-12684
                 URL: https://issues.apache.org/jira/browse/HADOOP-12684
             Project: Hadoop Common
          Issue Type: Wish
         Environment: Hadoop 2.6
            Reporter: william zhu

CombieFileInputFormat add blocks into one split use below codes : 
 if (maxSize != 0 && curSplitSize >= maxSize) {
            // create an input split and add it to the splits array
            addCreatedSplit(splits, nodes, validBlocks);
            curSplitSize = 0;
but , in some cases , the file is not splitable. Then files may contains both large and small
. In this case,  one split will be added a large file and then reach the  threshold  . 
I think some algorithm such as sort first will make the splits more even . 

This message was sent by Atlassian JIRA

View raw message