hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
Date Tue, 03 Oct 2017 23:02:00 GMT
Sahil Takiar created HIVE-17684:
-----------------------------------

             Summary: HoS memory issues with MapJoinMemoryExhaustionHandler
                 Key: HIVE-17684
                 URL: https://issues.apache.org/jira/browse/HIVE-17684
             Project: Hive
          Issue Type: Bug
          Components: Spark
            Reporter: Sahil Takiar
            Assignee: Sahil Takiar


We have seen a number of memory issues due the {{HashSinkOperator}} use of the {{MapJoinMemoryExhaustionHandler}}.
This handler is meant to detect scenarios where the small table is taking too much space in
memory, in which case a {{MapJoinMemoryExhaustionError}} is thrown.

The configs to control this logic are:

{{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
{{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)

The handler works by using the {{MemoryMXBean}} and uses the following logic to estimate how
much memory the {{HashMap}} is consuming: {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
MemoryMXBean#getHeapMemoryUsage().getMax()}}

The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be inaccurate. The value
returned by this method returns all reachable and unreachable memory on the heap, so there
may be a bunch of garbage data, and the JVM just hasn't taken the time to reclaim it all.
This can lead to intermittent failures of this check even though a simple GC would have reclaimed
enough space for the process to continue working.

We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. In Hive-on-MR
this probably made sense to use because every Hive task was run in a dedicated container,
so a Hive Task could assume it created most of the data on the heap. However, in Hive-on-Spark
there can be multiple Hive Tasks running in a single executor, each doing different things.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message