Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Message-Id: <201612130156.uBD1uT2u021772@ip-10-146-233-104.ec2.internal>
Date: Tue, 13 Dec 2016 01:56:29 +0000
From: "Impala Public Jenkins (Code Review)" <gerrit@cloudera.org>
To: Sailesh Mukil <sailesh@cloudera.com>, impala-cr@cloudera.com,
        reviews@impala.incubator.apache.org
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-4611=3A_Checking_perms_on_S3_files_is_a_very_expensive_no-op=0A?=
In-Reply-To: <gerrit.1481325678000.If9d1072c185a6162727019cdf1cb34d7f3f1c75c@gerrit.cloudera.org>
References: <gerrit.1481325678000.If9d1072c185a6162727019cdf1cb34d7f3f1c75c@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Tue, 13 Dec 2016 01:56:34 -0000

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-4611: Checking perms on S3 files is a very expensive no-op
......................................................................


IMPALA-4611: Checking perms on S3 files is a very expensive no-op

We call getPermissions() on partition directories to find out if
Impala has access to those files. On S3, this currently is a no-op
as the S3A connector does not try to set/get the permissions for S3
objects. So, it always returns the default set of permissions -> 777.
However, it still makes a roundtrip to S3 causing a slow down in the
Catalog.

We can return the READ_WRITE permission immediately if we know we are
accessing an S3 file, thereby avoiding the round trip to S3 for every
partition. This will greatly speedup metadata operations for S3 tables
and partitions, which is already known to be a big bottleneck.

If and when the S3A connector is able to manage permissions in
the future, we need to revisit this code. However, as permissions on
S3 are unsupported by Impala right now, we might as well gain on perf.

Change-Id: If9d1072c185a6162727019cdf1cb34d7f3f1c75c
Reviewed-on: http://gerrit.cloudera.org:8080/5449
Reviewed-by: Sailesh Mukil <sailesh@cloudera.com>
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
1 file changed, 10 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Sailesh Mukil: Looks good to me, approved


-- 
To view, visit http://gerrit.cloudera.org:8080/5449
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If9d1072c185a6162727019cdf1cb34d7f3f1c75c
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>