Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2670C200C2A for ; Wed, 15 Feb 2017 01:45:34 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 235D4160B6A; Wed, 15 Feb 2017 00:45:34 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6B92B160B5F for ; Wed, 15 Feb 2017 01:45:33 +0100 (CET) Received: (qmail 12989 invoked by uid 500); 15 Feb 2017 00:45:32 -0000 Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@impala.incubator.apache.org Received: (qmail 12978 invoked by uid 99); 15 Feb 2017 00:45:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Feb 2017 00:45:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0935118819F for ; Wed, 15 Feb 2017 00:45:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.363 X-Spam-Level: X-Spam-Status: No, score=0.363 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GAdDVfmwiIzF for ; Wed, 15 Feb 2017 00:45:31 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 262C45F659 for ; Wed, 15 Feb 2017 00:45:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id v1F0jL1J022408; Wed, 15 Feb 2017 00:45:21 GMT Date: Wed, 15 Feb 2017 00:45:21 +0000 From: "Bharath Vissapragada (Code Review)" To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org Message-ID: Reply-To: bharathv@cloudera.com X-Gerrit-MessageType: newchange Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-4840=3A_Fix_REFRESH_performance_regression=2E=0A?= X-Gerrit-Change-Id: I859b9fe93563ba886d0b5db6db42a14c88caada8 X-Gerrit-ChangeURL: X-Gerrit-Commit: e4db14950591133b6874e2d31e8daa23b8b67879 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 archived-at: Wed, 15 Feb 2017 00:45:34 -0000 Bharath Vissapragada has uploaded a new change for review. http://gerrit.cloudera.org:8080/6009 Change subject: IMPALA-4840: Fix REFRESH performance regression. ...................................................................... IMPALA-4840: Fix REFRESH performance regression. The fix for IMPALA-4172 introduced a regression in performance of the REFRESH command. The regression stems from the fact that we reload the block metadata of every valid data file without considering whether it has changed since the last load. This caused unnecessary metadata loads for unchanged files and thus increasing the runtime. The fix involves having the refresh codepath (and other operations that use the same codepath like insert etc.) to reload the metadata of only modified files by doing a listStatus() on the partition directory and checking the last modified time of each file. The initial/invalidate metadata load still fetches the block locations in bulk. Change-Id: I859b9fe93563ba886d0b5db6db42a14c88caada8 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java 2 files changed, 106 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/6009/1 -- To view, visit http://gerrit.cloudera.org:8080/6009 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I859b9fe93563ba886d0b5db6db42a14c88caada8 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Bharath Vissapragada