hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "william zhu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-12684) CombineFileInputFormat may cause splits uneveness
Date Thu, 31 Dec 2015 01:59:49 GMT
william zhu created HADOOP-12684:
------------------------------------

             Summary: CombineFileInputFormat may cause splits uneveness
                 Key: HADOOP-12684
                 URL: https://issues.apache.org/jira/browse/HADOOP-12684
             Project: Hadoop Common
          Issue Type: Wish
         Environment: Hadoop 2.6
            Reporter: william zhu


CombieFileInputFormat add blocks into one split use below codes : 
```
 if (maxSize != 0 && curSplitSize >= maxSize) {
            // create an input split and add it to the splits array
            addCreatedSplit(splits, nodes, validBlocks);
            curSplitSize = 0;
            validBlocks.clear();
          }
```
but , in some cases , the file is not splitable. Then files may contains both large and small
. In this case,  one split will be added a large file and then reach the  threshold  . 
I think some algorithm such as sort first will make the splits more even . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message