hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuxing Yao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-15290) Stripe size smaller than specified.
Date Mon, 28 Nov 2016 08:20:58 GMT
Yuxing Yao created HIVE-15290:
---------------------------------

             Summary: Stripe size smaller than specified.
                 Key: HIVE-15290
                 URL: https://issues.apache.org/jira/browse/HIVE-15290
             Project: Hive
          Issue Type: Bug
          Components: ORC
    Affects Versions: 2.0.1, 2.1.0, 2.0.0, 1.2.1, 1.2.0
            Reporter: Yuxing Yao


In Hive-1.2.0, the real stripe size of output orc file will be very small if most of table
data are empty, result in too many Column Statistics objects consumes most of the memory.
I found it become better in Hive-2.0.1, but the stripe size still much smaller than expected.
I saw there's a Jira item: https://issues.apache.org/jira/browse/HIVE-13232 moved the compressed
= null out of if block, this changes helps a lot, but for completely fix this, another change
is needed in `OutStream.getBufferSize()`
I've created the PR:
https://github.com/apache/hive/pull/118
Please take a look.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message