Mailing-List: contact dev-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flume.apache.org
Date: Mon, 11 Sep 2017 19:26:00 +0000 (UTC)
From: "Denes Arvay (JIRA)" <jira@apache.org>
To: dev@flume.apache.org
Message-ID: <JIRA.13101283.1505140336000.82347.1505157960266@Atlassian.JIRA>
In-Reply-To: <JIRA.13101283.1505140336000@Atlassian.JIRA>
References: <JIRA.13101283.1505140336000@Atlassian.JIRA> <JIRA.13101283.1505140336975@jira-lw-us.apache.org>
Subject: [jira] [Commented] (FLUME-3174) HdfsSink AWS S3A authentication
 does not work on JDK 8
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 11 Sep 2017 19:26:08 -0000


    [ https://issues.apache.org/jira/browse/FLUME-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161828#comment-16161828 ] 

Denes Arvay commented on FLUME-3174:
------------------------------------

Thanks for reporting and investigating this [~marcellhegedus].
Although you set the priority to "minor" I think this is more like a blocker for the 1.8 release (or at least critical), what do you think?
We should move forward with bumping the joda-time version, I'm linking this ticket with FLUME-3173 (Upgrade joda-time).


> HdfsSink AWS S3A authentication does not work on JDK 8
> ------------------------------------------------------
>
>                 Key: FLUME-3174
>                 URL: https://issues.apache.org/jira/browse/FLUME-3174
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Marcell Hegedus
>            Priority: Minor
>
> Flume writing to S3A with the following Hdfs Sink configuration fails with AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden...
> {code}
> a1.sinks = k1
> a1.sinks.k1.channel = c1
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.path = s3a://testflume/logs
> {code}
> AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided either in flume-env or in core-site.xml and running "hdfs dfs -ls s3a://testflume/logs" works properly.
> The cause and the fix is documented in [hadoop-aws/index.md|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#authentication-failures-when-running-on-java-8u60]
> {quote}A change in the Java 8 JVM broke some of the toString() string generation of Joda Time 2.8.0, which stopped the Amazon S3 client from being able to generate authentication headers suitable for validation by S3.
> Fix: Make sure that the version of Joda Time is 2.8.1 or later, or use a new version of Java 8.{quote}
> Tested that authentication is successful with
> * JDK 7
> * JDK 8 + joda-time updated to v2.9.6.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)