From issues-return-153108-archive-asf-public=cust-asf.ponee.io@hive.apache.org Mon Mar 18 07:20:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 15501180651 for ; Mon, 18 Mar 2019 08:20:03 +0100 (CET) Received: (qmail 97097 invoked by uid 500); 18 Mar 2019 07:20:03 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 97079 invoked by uid 99); 18 Mar 2019 07:20:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Mar 2019 07:20:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id AAF38C8945 for ; Mon, 18 Mar 2019 07:20:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Tg7_XWr8eQnP for ; Mon, 18 Mar 2019 07:20:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 29F8E5FCEE for ; Mon, 18 Mar 2019 07:20:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 9E303E20C5 for ; Mon, 18 Mar 2019 07:20:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 616B62459A for ; Mon, 18 Mar 2019 07:20:00 +0000 (UTC) Date: Mon, 18 Mar 2019 07:20:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214608&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214608 ] ASF GitHub Bot logged work on HIVE-21446: ----------------------------------------- Author: ASF GitHub Bot Created on: 18/Mar/19 07:19 Start Date: 18/Mar/19 07:19 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #569: HIVE-21446 : Hive Server going OOM during hive external table replications URL: https://github.com/apache/hive/pull/569#discussion_r266316160 ########## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java ########## @@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path sourcePath) throws IOE return createdDir; } - private boolean setTargetPathOwner(Path targetPath, Path sourcePath, String distCpDoAsUser) - throws IOException { - if (distCpDoAsUser == null) { + private boolean setTargetPathOwner(Path targetPath, Path sourcePath, UserGroupInformation proxyUser) + throws IOException, InterruptedException { + if (proxyUser == null) { return createAndSetPathOwner(targetPath, sourcePath); } - UserGroupInformation proxyUser = UserGroupInformation.createProxyUser( - distCpDoAsUser, UserGroupInformation.getLoginUser()); - try { - Path finalTargetPath = targetPath; - Path finalSourcePath = sourcePath; - return proxyUser.doAs((PrivilegedExceptionAction) () -> - createAndSetPathOwner(finalTargetPath, finalSourcePath)); - } catch (InterruptedException e) { - throw new IOException(e); + return proxyUser.doAs((PrivilegedExceptionAction) () -> + createAndSetPathOwner(targetPath, sourcePath)); + } + + private boolean checkIfPathExist(Path sourcePath, UserGroupInformation proxyUser) throws Exception { + if (proxyUser == null) { + return sourcePath.getFileSystem(conf).exists(sourcePath); } + return proxyUser.doAs((PrivilegedExceptionAction) () -> + sourcePath.getFileSystem(conf).exists(sourcePath)); } - private int handleException(Exception e, Path sourcePath, Path targetPath, int currentRetry) { + private int handleException(Exception e, Path sourcePath, Path targetPath, + int currentRetry, UserGroupInformation proxyUser) { try { - if (!sourcePath.getFileSystem(conf).exists(sourcePath)) { - LOG.warn("Source path missing " + sourcePath, e); + LOG.warn("Checking if source path " + sourcePath + " is missing for exception ", e); Review comment: change to info ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 214608) Time Spent: 1h 40m (was: 1.5h) > Hive Server going OOM during hive external table replications > ------------------------------------------------------------- > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl > Affects Versions: 4.0.0 > Reporter: mahesh kumar behera > Assignee: mahesh kumar behera > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)