From issues-return-142181-archive-asf-public=cust-asf.ponee.io@hive.apache.org Thu Nov 8 08:44:48 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E32A2180671 for ; Thu, 8 Nov 2018 08:44:47 +0100 (CET) Received: (qmail 76633 invoked by uid 500); 8 Nov 2018 07:44:46 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 76619 invoked by uid 99); 8 Nov 2018 07:44:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Nov 2018 07:44:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 38A3918E004 for ; Thu, 8 Nov 2018 07:44:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -107.5 X-Spam-Level: X-Spam-Status: No, score=-107.5 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Z4E6X9wGahu9 for ; Thu, 8 Nov 2018 07:44:43 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 600BE65A83 for ; Thu, 8 Nov 2018 07:35:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id AF9B2E2618 for ; Thu, 8 Nov 2018 07:35:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 8F6B1266BA for ; Thu, 8 Nov 2018 07:35:00 +0000 (UTC) Date: Thu, 8 Nov 2018 07:35:00 +0000 (UTC) From: "Hive QA (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-20682?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1667= 9384#comment-16679384 ]=20 Hive QA commented on HIVE-20682: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12947256/HIVE-20682.06.pat= ch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15526 tests ex= ecuted *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2] (batc= hId=3D85) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14804/test= Report Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14804/co= nsole Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14804/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12947256 - PreCommit-HIVE-Build > Async query execution can potentially fail if shared sessionHive is close= d by master thread. > -------------------------------------------------------------------------= ------------------- > > Key: HIVE-20682 > URL: https://issues.apache.org/jira/browse/HIVE-20682 > Project: Hive > Issue Type: Bug > Components: HiveServer2 > Affects Versions: 3.1.0, 4.0.0 > Reporter: Sankar Hariappan > Assignee: Sankar Hariappan > Priority: Major > Labels: pull-request-available > Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, HIVE-20682= .03.patch, HIVE-20682.04.patch, HIVE-20682.05.patch, HIVE-20682.06.patch > > > *Problem=C2=A0description:* > The master thread initializes the *sessionHive* object in *HiveSessionImp= l* class=C2=A0when=C2=A0we open=C2=A0a new session for a client connection = and by default all queries from this=C2=A0connection=C2=A0shares the same s= essionHive object.=C2=A0 > If the master thread executes a=C2=A0*synchronous*=C2=A0query, it closes = the sessionHive object (referred via thread local hiveDb) if=C2=A0 {{Hive.i= sCompatible}} returns false and sets new Hive object in thread local HiveDb= but doesn't=C2=A0change the sessionHive object in the session. Whereas, *a= synchronous* query execution via async=C2=A0threads=C2=A0never closes the s= essionHive object and it just creates a new one if needed and sets it as th= eir thread local hiveDb. > So, the problem can happen=C2=A0in the case where=C2=A0an=C2=A0*asynchron= ous* query is being=C2=A0executed by async threads refers to sessionHive ob= ject and the master thread receives a *synchronous* query=C2=A0that=C2=A0cl= oses=C2=A0the same=C2=A0sessionHive object.=C2=A0 > Also, each query execution overwrites the thread local hiveDb object to s= essionHive object which potentially leaks a metastore connection if the pre= vious synchronous query execution re-created the Hive object. > *Possible Fix:* > The *sessionHive* object could be shared my multiple threads and so it sh= ouldn't be allowed to be closed by any query execution threads when they re= -create the Hive object due to changes in Hive configurations. But the Hive= objects created by query execution threads should be closed when the threa= d exits. > So, it is proposed to have an *isAllowClose*=C2=A0flag (default: *true*) = in Hive object which should be set to *false* for=C2=A0*sessionHive* and wo= uld be forcefully closed when the session is closed or released. > Also, when we reset *sessionHive* object with new one due to changes in *= sessionConf*, the old one should be closed when no async thread is referrin= g to it. This can be done using "*finalize*" method of Hive object where we= can close HMS connection when Hive object is garbage collected. > cc [~pvary] -- This message was sent by Atlassian JIRA (v7.6.3#76005)