flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Processing S3 data with Apache Flink
Date Tue, 20 Oct 2015 13:34:13 GMT
@Konstantin (2) : Can you try the workaround described by Robert, with the
"s3n" file system scheme?

We are removing the custom S3 connector now, simply reusing Hadoop's S3
connector for all cases.

@Kostia:
You are right, there should be no broken stuff that is not clearly marked
as "beta". For the S3 connector, that was a problem in the testing on our
side and should not have happened.
In general, you can assume that stuff in "flink-contrib" is in beta status,
as well as the stuff in "flink-staging" (although much of the staging stuff
will graduate with the next release). All code not in these projects should
be well functioning. We test a lot, so there should be not many broken
cases like that S3 connector.

Greetings,
Stephan


On Wed, Oct 14, 2015 at 11:44 AM, Ufuk Celebi <uce@apache.org> wrote:

>
> > On 10 Oct 2015, at 22:59, snntr <konstantin.knauf@tngtech.com> wrote:
> >
> > Hey everyone,
> >
> > I was having the same problem with S3 and found this thread very useful.
> > Everything works fine now, when I start Flink from my IDE, but when I run
> > the jar in local mode I keep getting
> >
> > java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access
> Key
> > must be specified as the username or password (respectively) of a s3n
> URL,
> > or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
> > properties (respectively).
> >
> > I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
> > machine with the required properties. What am I missing?
> >
> > Any advice is highly appreciated ;)
>
> This looks like a problem with picking up the Hadoop config. Can you look
> into the logs to check whether the configuration is picked up? Change the
> log settings to DEBUG in log/log4j.properties for this. And can you provide
> the complete stack trace?
>
> – Ufuk
>
>

Mime
View raw message