hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15503) LLAP: Fix use of Runtime.getRuntime.maxMemory in Hive operators
Date Fri, 23 Dec 2016 04:03:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771818#comment-15771818
] 

Prasanth Jayachandran commented on HIVE-15503:
----------------------------------------------

[~hagleitn] The patch is ready for review now. I am still working on the tests. The current
test flushes after every row because memory usage shows high compared to max value. I will
update the patch after identifying the root cause. 

> LLAP: Fix use of Runtime.getRuntime.maxMemory in Hive operators
> ---------------------------------------------------------------
>
>                 Key: HIVE-15503
>                 URL: https://issues.apache.org/jira/browse/HIVE-15503
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Critical
>         Attachments: HIVE-15503.1.patch, HIVE-15503.WIP.patch
>
>
> {code}
> ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java:    maxHashTblMemory
= (long) (memoryPercentage * Runtime.getRuntime().maxMemory());
> ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java:    // Total Free Memory = maxMemory()
- Used Memory;
> ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java:    long totalFreeMemory = Runtime.getRuntime().maxMemory()
-
> {code}
> This will not work very well with LLAP because of the memory sharing by executors. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message