hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Haindrich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
Date Mon, 21 Aug 2017 10:37:02 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135008#comment-16135008
] 

Zoltan Haindrich commented on HIVE-17344:
-----------------------------------------

[~sershe] I think it is currently true that remaining == capacity...but the underlying OrcTail
resets the position to 0 when it reads explicitly...so if the bb position is not at the beginning
remainig() would could underestimate the memory usage...so I think using capacity would be
better...because in that case it may only overestimate memory usage...and not under (but there
is the case when someone uses windows...)

I haven't seen any references where a bigger bb was sliced and passed to this function...if
that happens...the capacity of the sliced bb is the old limit...which is ok...

I agree that currently capacity/remaining will do the same...but capacity would be more explicit

> LocalCache element memory usage is not calculated properly.
> -----------------------------------------------------------
>
>                 Key: HIVE-17344
>                 URL: https://issues.apache.org/jira/browse/HIVE-17344
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Janos Gub
>            Assignee: Janos Gub
>         Attachments: HIVE-17344.patch
>
>
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
>   return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus allowing this
cache have elements MAXWEIGHT/100 of arbitrary size. I think the correct solution would be
bb.capacity.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message