impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Laszlo Gaal (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-6067: Enable s3 access via IAM roles for EC2 VMs
Date Wed, 22 Nov 2017 21:04:24 GMT
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/8294 )

Change subject: IMPALA-6067: Enable s3 access via IAM roles for EC2 VMs
......................................................................


Patch Set 3:

>[...]Please do answer the question about changing
 > the FE tests to just use s3a so that we can remove the hackery
 > altogether.[...]

In short: it depends on your end goal.
If the end goal is to get rid of the s3n: provider and its supporting menagerie (jets3t, etc.),
then yes, it can be done with a small price: a single negative test in AnalyzeStmtsTest.java
will have to go. The code there in https://github.com/cloudera/Impala/blob/cdh5-trunk/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java#L3000
performs a test step that should fail for s3n -- but if I remove the s3n properties from core-site.xml,
this line blows up complaining about missing credentials. The stack trace reveals that Impala's
HdfsUri::analyze() calls into Hadoop's Path::getFileSystem() to find out what kind of filesystem
the URI points to; this in turn calls down all the way to org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:74)
being called from Jets3t.

If your end goal is to remove dummy credentials from core-site.xml.tmpl, this will probably
not be complete success: the s3a: provider behaves in a similar () though not identical) way.
When the front-end tests are run without an S3 container in sight (e.g. your local dev box
with no S3 access set up), AnalyzeDDLTest's s3a: URIs still require similar dummy credentials,
but with s3a: these can be supplied in environment variables. This means that core-site.xml[.tmpl]
could be cleaned up, but dummy credentials in envireonment variables would still be there.


-- 
To view, visit http://gerrit.cloudera.org:8080/8294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I14cd9d4453a91baad3c379aa7e4944993fca95ae
Gerrit-Change-Number: 8294
Gerrit-PatchSet: 3
Gerrit-Owner: Laszlo Gaal <laszlo.gaal@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: Jim Apple <jbapple-impala@apache.org>
Gerrit-Reviewer: Joe McDonnell <joemcdonnell@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <laszlo.gaal@cloudera.com>
Gerrit-Reviewer: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <philip@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Comment-Date: Wed, 22 Nov 2017 21:04:24 +0000
Gerrit-HasComments: No

Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message