hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11807) Set ORC buffer size in relation to set stripe size
Date Wed, 21 Oct 2015 20:29:27 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967823#comment-14967823
] 

Prasanth Jayachandran commented on HIVE-11807:
----------------------------------------------

[~owen.omalley] Can you please rebase your patch to trunk?

> Set ORC buffer size in relation to set stripe size
> --------------------------------------------------
>
>                 Key: HIVE-11807
>                 URL: https://issues.apache.org/jira/browse/HIVE-11807
>             Project: Hive
>          Issue Type: Improvement
>          Components: File Formats
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: HIVE-11807.patch, HIVE-11807.patch
>
>
> A customer produced ORC files with very small stripe sizes (10k rows/stripe) by setting
a small 64MB stripe size and 256K buffer size for a 54 column table. At that size, each of
the streams only get a buffer or two before the stripe size is reached. The current code uses
the available memory instead of the stripe size and thus doesn't shrink the buffer size if
the JVM has much more memory than the stripe size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message