flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Denes Arvay (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3174) HdfsSink AWS S3A authentication does not work on JDK 8
Date Mon, 11 Sep 2017 19:26:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161828#comment-16161828

Denes Arvay commented on FLUME-3174:

Thanks for reporting and investigating this [~marcellhegedus].
Although you set the priority to "minor" I think this is more like a blocker for the 1.8 release
(or at least critical), what do you think?
We should move forward with bumping the joda-time version, I'm linking this ticket with FLUME-3173
(Upgrade joda-time).

> HdfsSink AWS S3A authentication does not work on JDK 8
> ------------------------------------------------------
>                 Key: FLUME-3174
>                 URL: https://issues.apache.org/jira/browse/FLUME-3174
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Marcell Hegedus
>            Priority: Minor
> Flume writing to S3A with the following Hdfs Sink configuration fails with AmazonS3Exception:
Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden...
> {code}
> a1.sinks = k1
> a1.sinks.k1.channel = c1
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.path = s3a://testflume/logs
> {code}
> AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided either in flume-env or in core-site.xml
and running "hdfs dfs -ls s3a://testflume/logs" works properly.
> The cause and the fix is documented in [hadoop-aws/index.md|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#authentication-failures-when-running-on-java-8u60]
> {quote}A change in the Java 8 JVM broke some of the toString() string generation of Joda
Time 2.8.0, which stopped the Amazon S3 client from being able to generate authentication
headers suitable for validation by S3.
> Fix: Make sure that the version of Joda Time is 2.8.1 or later, or use a new version
of Java 8.{quote}
> Tested that authentication is successful with
> * JDK 7
> * JDK 8 + joda-time updated to v2.9.6.

This message was sent by Atlassian JIRA

View raw message