impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Laszlo Gaal (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-6067: Enable S3 access via IAM roles for EC2 VMs
Date Tue, 05 Dec 2017 23:38:36 GMT
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/8294 )

Change subject: IMPALA-6067: Enable S3 access via IAM roles for EC2 VMs
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8294/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:

http://gerrit.cloudera.org:8080/#/c/8294/4/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@a3000
PS4, Line 3000: 
> Can we restore this?  LOAD DATA INPATH doesn't get coverage in the other te
I'm facing a problem with this one.
Note, that the original test here is a negative one: it verifies that LOAD DATA INPATH rejects
an s3n: filesystem reference.
With s3n dummy credentials removed from the minicluster configuration and Hadoop3 removing
the s3n provider completely the negative test will still fail, but with a different error/exception,
and the test would not be relevant any more.

I tried to change it to a positive test using AnalyzesOk() and an s3a: reference. The problem
there was that the s3a: provider seems to insist on hitting the S3 object URI encoded in the
test. That means that a successful test step needs an existing S3 bucket and object ("file")
and valid credentials to access the file. I think it would be problematic if FE unit tests
suddenly started requiring valid S3 credentials.
I could make the S3 bucket world-readable, but it still feels awkward.

Also, the end2end tests actually test various LOAD DATA INPATH scenarios, e.g. https://github.com/cloudera/Impala/blob/cdh5-trunk/testdata/workloads/functional-query/queries/QueryTest/load.test#L35
These tests use the FILESYSTEM_PREFIX environment variable, so they run against S3 (specifically
s3a:) when the tests are run on S3.

Would you be comfortable with disabling this test step and following it up in a separate Jira?



-- 
To view, visit http://gerrit.cloudera.org:8080/8294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I14cd9d4453a91baad3c379aa7e4944993fca95ae
Gerrit-Change-Number: 8294
Gerrit-PatchSet: 4
Gerrit-Owner: Laszlo Gaal <laszlo.gaal@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: Jim Apple <jbapple-impala@apache.org>
Gerrit-Reviewer: Joe McDonnell <joemcdonnell@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <laszlo.gaal@cloudera.com>
Gerrit-Reviewer: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <philip@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Zach Amsden <zamsden@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Dec 2017 23:38:36 +0000
Gerrit-HasComments: Yes

Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message