hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-929) hive.map.mergefiles increases the size in some cases
Date Fri, 13 Nov 2009 02:44:39 GMT

    [ https://issues.apache.org/jira/browse/HIVE-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777348#action_12777348
] 

Namit Jain commented on HIVE-929:
---------------------------------

Currently, we use only one size:

"hive.merge.size.per.task"

whose default value is 256M.

We should add another parameter

"hive.merge.smallfiles.avgsize"

whose default value can be much smaller, say 16M.

We will only merge if the current average size of a file < "hive.merge.smallfiles.avgsize".

This will make sure that merging will happen only in very bad cases.

> hive.map.mergefiles increases the size in some cases
> ----------------------------------------------------
>
>                 Key: HIVE-929
>                 URL: https://issues.apache.org/jira/browse/HIVE-929
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Namit Jain
>
> Due to random clustering, the size is increased in some cases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message