Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 3 Oct 2017 23:02:00 +0000 (UTC)
From: "Sahil Takiar (JIRA)" <jira@apache.org>
To: dev@hive.apache.org
Message-ID: <JIRA.13106829.1507071672000.265318.1507071720165@Atlassian.JIRA>
In-Reply-To: <JIRA.13106829.1507071672000@Atlassian.JIRA>
References: <JIRA.13106829.1507071672000@Atlassian.JIRA> <JIRA.13106829.1507071672162@jira-lw-us.apache.org>
Subject: [jira] [Created] (HIVE-17684) HoS memory issues with
 MapJoinMemoryExhaustionHandler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 03 Oct 2017 23:02:04 -0000

Sahil Takiar created HIVE-17684:
-----------------------------------

             Summary: HoS memory issues with MapJoinMemoryExhaustionHandler
                 Key: HIVE-17684
                 URL: https://issues.apache.org/jira/browse/HIVE-17684
             Project: Hive
          Issue Type: Bug
          Components: Spark
            Reporter: Sahil Takiar
            Assignee: Sahil Takiar


We have seen a number of memory issues due the {{HashSinkOperator}} use of the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect scenarios where the small table is taking too much space in memory, in which case a {{MapJoinMemoryExhaustionError}} is thrown.

The configs to control this logic are:

{{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
{{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)

The handler works by using the {{MemoryMXBean}} and uses the following logic to estimate how much memory the {{HashMap}} is consuming: {{MemoryMXBean#getHeapMemoryUsage().getUsed() / MemoryMXBean#getHeapMemoryUsage().getMax()}}

The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be inaccurate. The value returned by this method returns all reachable and unreachable memory on the heap, so there may be a bunch of garbage data, and the JVM just hasn't taken the time to reclaim it all. This can lead to intermittent failures of this check even though a simple GC would have reclaimed enough space for the process to continue working.

We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS. In Hive-on-MR this probably made sense to use because every Hive task was run in a dedicated container, so a Hive Task could assume it created most of the data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks running in a single executor, each doing different things.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)