impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Laszlo Gaal (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-6067: Enable s3 access via IAM roles for EC2 VMs
Date Wed, 22 Nov 2017 21:04:24 GMT
Laszlo Gaal has posted comments on this change. ( )

Change subject: IMPALA-6067: Enable s3 access via IAM roles for EC2 VMs

Patch Set 3:

>[...]Please do answer the question about changing
 > the FE tests to just use s3a so that we can remove the hackery
 > altogether.[...]

In short: it depends on your end goal.
If the end goal is to get rid of the s3n: provider and its supporting menagerie (jets3t, etc.),
then yes, it can be done with a small price: a single negative test in
will have to go. The code there in
performs a test step that should fail for s3n -- but if I remove the s3n properties from core-site.xml,
this line blows up complaining about missing credentials. The stack trace reveals that Impala's
HdfsUri::analyze() calls into Hadoop's Path::getFileSystem() to find out what kind of filesystem
the URI points to; this in turn calls down all the way to org.apache.hadoop.fs.s3.S3Credentials.initialize(
being called from Jets3t.

If your end goal is to remove dummy credentials from core-site.xml.tmpl, this will probably
not be complete success: the s3a: provider behaves in a similar () though not identical) way.
When the front-end tests are run without an S3 container in sight (e.g. your local dev box
with no S3 access set up), AnalyzeDDLTest's s3a: URIs still require similar dummy credentials,
but with s3a: these can be supplied in environment variables. This means that core-site.xml[.tmpl]
could be cleaned up, but dummy credentials in envireonment variables would still be there.

To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I14cd9d4453a91baad3c379aa7e4944993fca95ae
Gerrit-Change-Number: 8294
Gerrit-PatchSet: 3
Gerrit-Owner: Laszlo Gaal <>
Gerrit-Reviewer: Alex Behm <>
Gerrit-Reviewer: David Knupp <>
Gerrit-Reviewer: Jim Apple <>
Gerrit-Reviewer: Joe McDonnell <>
Gerrit-Reviewer: Lars Volker <>
Gerrit-Reviewer: Laszlo Gaal <>
Gerrit-Reviewer: Michael Brown <>
Gerrit-Reviewer: Philip Zeyliger <>
Gerrit-Reviewer: Sailesh Mukil <>
Gerrit-Reviewer: Tim Armstrong <>
Gerrit-Comment-Date: Wed, 22 Nov 2017 21:04:24 +0000
Gerrit-HasComments: No

  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message