hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Pivovarov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10711) Tez HashTableLoader attempts to allocate more memory than available when HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD exceeds process max mem
Date Fri, 15 May 2015 19:04:02 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546007#comment-14546007
] 

Alexander Pivovarov commented on HIVE-10711:
--------------------------------------------

1. Lets say noConditionalTaskThreshold is 3990MB and processMaxMemory = 4000MB
In this case noConditionalTaskThreshold adjustment will be skipped.

But if we slightly increase noConditionalTaskThreshold to lets say 4010 MB then your code
will work and will change noConditionalTaskThreshold to 2000 MB

20 MB change in noConditionalTaskThreshold makes significant change in final value - 3990
MB vs 2000 MB  (which is a big difference)

I think that noConditionalTaskThreshold adjustment should adjust noConditionalTaskThreshold
to smth which is close to 3990 MB

This is why I think using HIVEHASHTABLEMAXMEMORYUSAGE=0.9 instead of 0.5 makes sense.

2. To improve the algo even more we can check if noConditionalTaskThreshold is bigger than
90% of processMaxMemory and adjust it
{code}
if (noConditionalTaskThreshold / processMaxMemory > HIVEHASHTABLEMAXMEMORYUSAGE) {
  noConditionalTaskThreshold = processMaxMemory * HIVEHASHTABLEMAXMEMORYUSAGE
}
{code}
What you think?

> Tez HashTableLoader attempts to allocate more memory than available when HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD
exceeds process max mem
> ------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-10711
>                 URL: https://issues.apache.org/jira/browse/HIVE-10711
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Jason Dere
>            Assignee: Jason Dere
>         Attachments: HIVE-10711.1.patch, HIVE-10711.2.patch
>
>
> Tez HashTableLoader bases its memory allocation on HIVECONVERTJOINNOCONDITIONALTASKTHRESHOLD.
If this value is largeer than the process max memory then this can result in the HashTableLoader
trying to use more memory than available to the process.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message