hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-929) hive.map.mergefiles increases the size in some cases
Date Fri, 13 Nov 2009 02:44:39 GMT

    [ https://issues.apache.org/jira/browse/HIVE-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777348#action_12777348

Namit Jain commented on HIVE-929:

Currently, we use only one size:


whose default value is 256M.

We should add another parameter


whose default value can be much smaller, say 16M.

We will only merge if the current average size of a file < "hive.merge.smallfiles.avgsize".

This will make sure that merging will happen only in very bad cases.

> hive.map.mergefiles increases the size in some cases
> ----------------------------------------------------
>                 Key: HIVE-929
>                 URL: https://issues.apache.org/jira/browse/HIVE-929
>             Project: Hadoop Hive
>          Issue Type: Bug
>            Reporter: Namit Jain
> Due to random clustering, the size is increased in some cases.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message