flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Knauf <konstantin.kn...@tngtech.com>
Subject Re: Processing S3 data with Apache Flink
Date Sat, 21 Nov 2015 11:48:57 GMT
Hi Ufuk,

sorry for not getting back to you for so long, and thanks for your
answer. The problem persists unfortunately. Running the job from the IDE
works (with core-site.xml on classpath), running it in local standalone
mode does not. AccessKeyID and SecretAccesKey are not found.

Attached the jobmanager log on DEBUG level. The core-site.xml is
definitely at the configured location.

I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 to
run the jar in local mode. Do I have to use the Hadoop 2.x version for
this to work? I have put hadoop-common-2.3.jar into the flink lib folder.

I don't know if it is relevant (but it seems to be related), when I run
the job from my IDE I get the warning:

2015-11-21 12:43:11 WARN  NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable

Cheers and thank you,


On 14.10.2015 11:44, Ufuk Celebi wrote:
>> On 10 Oct 2015, at 22:59, snntr <konstantin.knauf@tngtech.com> wrote:
>> Hey everyone, 
>> I was having the same problem with S3 and found this thread very useful.
>> Everything works fine now, when I start Flink from my IDE, but when I run
>> the jar in local mode I keep getting 
>> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
>> must be specified as the username or password (respectively) of a s3n URL,
>> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
>> properties (respectively).
>> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
>> machine with the required properties. What am I missing?
>> Any advice is highly appreciated ;)
> This looks like a problem with picking up the Hadoop config. Can you look into the logs
to check whether the configuration is picked up? Change the log settings to DEBUG in log/log4j.properties
for this. And can you provide the complete stack trace?
> – Ufuk

Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

View raw message