hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15665) LLAP: OrcFileMetadata objects in cache can impact heap usage
Date Thu, 19 Jan 2017 21:21:26 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830629#comment-15830629
] 

Sergey Shelukhin commented on HIVE-15665:
-----------------------------------------

There's JIRA somewhere to move metadata cache off-heap, using the same allocator and storing
ORC buffers as is (or concatenated). That should be pretty simple design-wise.

> LLAP: OrcFileMetadata objects in cache can impact heap usage
> ------------------------------------------------------------
>
>                 Key: HIVE-15665
>                 URL: https://issues.apache.org/jira/browse/HIVE-15665
>             Project: Hive
>          Issue Type: Improvement
>          Components: llap
>            Reporter: Rajesh Balamohan
>
> OrcFileMetadata internally has filestats, stripestats etc which are allocated in heap.
On large data sets, this could have an impact on the heap usage and the memory usage by different
executors in LLAP.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message