flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Processing S3 data with Apache Flink
Date Sat, 21 Nov 2015 13:52:13 GMT
Hi,

It seems that you've set the "fs.hdfs.hadoopconf" configuration parameter
to a file. I think you have to set it the directory containing the
configuration.
Sorry, I know that's not very intuitive, but in Hadoop the settings for in
different files (hdfs|yarn|core)-site.xml.


On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf <
konstantin.knauf@tngtech.com> wrote:

> Hi Ufuk,
>
> sorry for not getting back to you for so long, and thanks for your
> answer. The problem persists unfortunately. Running the job from the IDE
> works (with core-site.xml on classpath), running it in local standalone
> mode does not. AccessKeyID and SecretAccesKey are not found.
>
> Attached the jobmanager log on DEBUG level. The core-site.xml is
> definitely at the configured location.
>
> I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 to
> run the jar in local mode. Do I have to use the Hadoop 2.x version for
> this to work? I have put hadoop-common-2.3.jar into the flink lib folder.
>
> I don't know if it is relevant (but it seems to be related), when I run
> the job from my IDE I get the warning:
>
> 2015-11-21 12:43:11 WARN  NativeCodeLoader:62 - Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
>
> Cheers and thank you,
>
> Konstantin
>
>
> On 14.10.2015 11:44, Ufuk Celebi wrote:
> >
> >> On 10 Oct 2015, at 22:59, snntr <konstantin.knauf@tngtech.com> wrote:
> >>
> >> Hey everyone,
> >>
> >> I was having the same problem with S3 and found this thread very useful.
> >> Everything works fine now, when I start Flink from my IDE, but when I
> run
> >> the jar in local mode I keep getting
> >>
> >> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access
> Key
> >> must be specified as the username or password (respectively) of a s3n
> URL,
> >> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
> >> properties (respectively).
> >>
> >> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local
> >> machine with the required properties. What am I missing?
> >>
> >> Any advice is highly appreciated ;)
> >
> > This looks like a problem with picking up the Hadoop config. Can you
> look into the logs to check whether the configuration is picked up? Change
> the log settings to DEBUG in log/log4j.properties for this. And can you
> provide the complete stack trace?
> >
> > – Ufuk
> >
> >
>
> --
> Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>

Mime
View raw message