Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 113D2200C89 for ; Sat, 20 May 2017 06:34:49 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0FAC6160BD2; Sat, 20 May 2017 04:34:49 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 55E79160BD1 for ; Sat, 20 May 2017 06:34:48 +0200 (CEST) Received: (qmail 70448 invoked by uid 500); 20 May 2017 04:34:47 -0000 Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@impala.incubator.apache.org Received: (qmail 70432 invoked by uid 99); 20 May 2017 04:34:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 May 2017 04:34:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C6D9D180533 for ; Sat, 20 May 2017 04:34:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 7Y-kY7IFUf_p for ; Sat, 20 May 2017 04:34:45 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 7A1565F20C for ; Sat, 20 May 2017 04:34:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id v4K4YiUl015621; Sat, 20 May 2017 04:34:44 GMT Message-Id: <201705200434.v4K4YiUl015621@ip-10-146-233-104.ec2.internal> Date: Sat, 20 May 2017 04:34:44 +0000 From: "Dimitris Tsirogiannis (Code Review)" To: Taras Bobrovytsky , impala-cr@cloudera.com, reviews@impala.incubator.apache.org CC: Bharath Vissapragada , Alex Behm Reply-To: dtsirogiannis@cloudera.com X-Gerrit-MessageType: comment Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-5259=3A_Add_REFRESH_FUNCTIONS_=3Cdb=3E_statement=0A?= X-Gerrit-Change-Id: I3625c88bb51cca833f3293c224d3f0feb00e6e0b X-Gerrit-ChangeURL: X-Gerrit-Commit: a6b1557fe6ea39e6d4097ec1cfe3287ed3270348 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 archived-at: Sat, 20 May 2017 04:34:49 -0000 Dimitris Tsirogiannis has posted comments on this change. Change subject: IMPALA-5259: Add REFRESH FUNCTIONS statement ...................................................................... Patch Set 7: (5 comments) http://gerrit.cloudera.org:8080/#/c/6878/4/fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java File fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java: PS4, Line 38: invalidating the entire catalog "or refreshing database functions." PS4, Line 110: toSql() Don't we need to update this function? http://gerrit.cloudera.org:8080/#/c/6878/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: PS7, Line 628: loadFunctionsFromDbParams(tmpDb, msDb); : // Load Java UDFs from HMS into the temporary db. : loadJavaFunctions(tmpDb, javaFns); Both these methods not only extract and load the functions into the target database but they also increment and acquire a new catalog version for each function. This is redundant and is actually performed again in L661. We should avoid this. PS7, Line 635: catalogLock_.writeLock().lock(); This lock doesn't protect against concurrent operations on databases, hence the block in L636-670 is not safe. If you look at the CatalogOpExecutor, add/drop db/functions are protected by the metastoreDdlLock_ (this lock variable is poorly named) and I believe this lock should be used here instead. http://gerrit.cloudera.org:8080/#/c/6878/7/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java File fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java: PS7, Line 256: // Remove the function first, in case there is an existing function with the same : // name and signature. : removeFunction(catalogObject.getFn(), catalogObject.getCatalog_version()); This is weird and not expected. Is this because addFunction doesn't replace the function if it already exists? If so, we should fix that and remove the removeFunction call. -- To view, visit http://gerrit.cloudera.org:8080/6878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3625c88bb51cca833f3293c224d3f0feb00e6e0b Gerrit-PatchSet: 7 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Taras Bobrovytsky Gerrit-HasComments: Yes