hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qiang.Kang (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-24311) Rowcontainer should reset readBlocks when we clear rows to prevent OOM.
Date Sat, 24 Oct 2020 07:40:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-24311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220022#comment-17220022
] 

Qiang.Kang commented on HIVE-24311:
-----------------------------------

Just added a patch for this issue. Anybody help to review the code?  Thanks!
 

> Rowcontainer should reset readBlocks when we clear rows to prevent OOM.
> -----------------------------------------------------------------------
>
>                 Key: HIVE-24311
>                 URL: https://issues.apache.org/jira/browse/HIVE-24311
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: All Versions
>            Reporter: Qiang.Kang
>            Assignee: Qiang.Kang
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Hi, We found that `Rowcontainer.clearRows()` only reset some index, such as: `addCursor`,
`itrCursor`, etc, without resetting read blocks.
> As we all know, `currentReadBlock` and `currentWriteBlock` is the main memory usage of
a `Rowcontainer` and might be very large, depending on the data pattern.
> `currentReadBlock` and `currentWriteBlock` won't be the same object after rowcontainer
flushed data to disk.
> Resetting `currentReadBlock` and `currentWriteBlock` while clearing rows will prevent
OOM.
>  
> Therefore, I submit a patch to reset read blocks for `Rowcontainer`, just like `PTFRowcontainer`
does.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message