Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 07926200B67 for ; Tue, 12 Jul 2016 02:13:13 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 04B0B160A78; Tue, 12 Jul 2016 00:13:13 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 554BB160A7D for ; Tue, 12 Jul 2016 02:13:12 +0200 (CEST) Received: (qmail 69581 invoked by uid 500); 12 Jul 2016 00:13:11 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 69567 invoked by uid 99); 12 Jul 2016 00:13:11 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2016 00:13:11 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 547932C02A4 for ; Tue, 12 Jul 2016 00:13:11 +0000 (UTC) Date: Tue, 12 Jul 2016 00:13:11 +0000 (UTC) From: "Thomas Friedrich (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (HIVE-14210) SSLFactory truststore reloader threads leaking in HiveServer2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 12 Jul 2016 00:13:13 -0000 [ https://issues.apache.org/jira/browse/HIVE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371944#comment-15371944 ] Thomas Friedrich edited comment on HIVE-14210 at 7/12/16 12:12 AM: ------------------------------------------------------------------- Provided patch for ExecDriver.java to call jobclient.close() was (Author: tfriedr): Patch for ExecDriver.java > SSLFactory truststore reloader threads leaking in HiveServer2 > ------------------------------------------------------------- > > Key: HIVE-14210 > URL: https://issues.apache.org/jira/browse/HIVE-14210 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 > Affects Versions: 1.2.1, 2.0.0, 2.1.0 > Reporter: Thomas Friedrich > Attachments: HIVE-14210.patch > > > We found an issue in a customer environment where the HS2 crashed after a few days and the Java core dump contained several thousands of truststore reloader threads: > "Truststore reloader thread" #126 daemon prio=5 os_prio=0 tid=0x00007f680d2e3000 nid=0x98fd waiting on > condition [0x00007f67e482c000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run > (ReloadingX509TrustManager.java:225) > at java.lang.Thread.run(Thread.java:745) > We found the issue to be caused by a bug in Hadoop where the TimelineClientImpl is not destroying the SSLFactory if SSL is enabled in Hadoop and the timeline server is running. I opened YARN-5309 which has more details on the problem, and a patch was submitted a few days back. > In addition to the changes in Hadoop, there are a couple of Hive changes required: > - ExecDriver needs to call jobclient.close() to trigger the clean-up of the resources after the submitted job is done/failed > - Hive needs to pick up a newer release of Hadoop to pick up MAPREDUCE-6618 and MAPREDUCE-6621 that fixed issues with calling jobclient.close(). Both fixes are included in Hadoop 2.6.4. > However, since we also need to pick up YARN-5309, we need to wait for a new release of Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)