flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Processing S3 data with Apache Flink
Date Tue, 20 Oct 2015 13:34:13 GMT
@Konstantin (2) : Can you try the workaround described by Robert, with the
"s3n" file system scheme?

We are removing the custom S3 connector now, simply reusing Hadoop's S3
connector for all cases.

You are right, there should be no broken stuff that is not clearly marked
as "beta". For the S3 connector, that was a problem in the testing on our
side and should not have happened.
In general, you can assume that stuff in "flink-contrib" is in beta status,
as well as the stuff in "flink-staging" (although much of the staging stuff
will graduate with the next release). All code not in these projects should
be well functioning. We test a lot, so there should be not many broken
cases like that S3 connector.


On Wed, Oct 14, 2015 at 11:44 AM, Ufuk Celebi <uce@apache.org> wrote:

> > On 10 Oct 2015, at 22:59, snntr <konstantin.knauf@tngtech.com> wrote:
> >
> > Hey everyone,
> >
> > I was having the same problem with S3 and found this thread very useful.
> > Everything works fine now, when I start Flink from my IDE, but when I run
> > the jar in local mode I keep getting
> >
> > java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access
> Key
> > must be specified as the username or password (respectively) of a s3n
> URL,
> > or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
> > properties (respectively).
> >
> > I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
> > machine with the required properties. What am I missing?
> >
> > Any advice is highly appreciated ;)
> This looks like a problem with picking up the Hadoop config. Can you look
> into the logs to check whether the configuration is picked up? Change the
> log settings to DEBUG in log/log4j.properties for this. And can you provide
> the complete stack trace?
> – Ufuk

View raw message