hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Haindrich (JIRA)" <>
Subject [jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
Date Mon, 21 Aug 2017 10:37:02 GMT


Zoltan Haindrich commented on HIVE-17344:

[~sershe] I think it is currently true that remaining == capacity...but the underlying OrcTail
resets the position to 0 when it reads if the bb position is not at the beginning
remainig() would could underestimate the memory I think using capacity would be
better...because in that case it may only overestimate memory usage...and not under (but there
is the case when someone uses windows...)

I haven't seen any references where a bigger bb was sliced and passed to this function...if
that happens...the capacity of the sliced bb is the old limit...which is ok...

I agree that currently capacity/remaining will do the same...but capacity would be more explicit

> LocalCache element memory usage is not calculated properly.
> -----------------------------------------------------------
>                 Key: HIVE-17344
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Janos Gub
>            Assignee: Janos Gub
>         Attachments: HIVE-17344.patch
> Orc footer cache has a calculation of memory usage:
> {code:java}
> public int getMemoryUsage() {
>   return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads (semi-arbitrary).
> }
> {code}
> ByteBuffer.remaining returns the remaining space in the bytebuffer, thus allowing this
cache have elements MAXWEIGHT/100 of arbitrary size. I think the correct solution would be

This message was sent by Atlassian JIRA

View raw message