Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A168F18230 for ; Sat, 21 Nov 2015 13:52:40 +0000 (UTC) Received: (qmail 58206 invoked by uid 500); 21 Nov 2015 13:52:35 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 58128 invoked by uid 500); 21 Nov 2015 13:52:35 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 58119 invoked by uid 99); 21 Nov 2015 13:52:35 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Nov 2015 13:52:35 +0000 Received: from mail-lf0-f45.google.com (mail-lf0-f45.google.com [209.85.215.45]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 95E8A1A04B3 for ; Sat, 21 Nov 2015 13:52:34 +0000 (UTC) Received: by lfaz4 with SMTP id z4so84668755lfa.0 for ; Sat, 21 Nov 2015 05:52:32 -0800 (PST) X-Received: by 10.25.18.92 with SMTP id h89mr3994093lfi.54.1448113952579; Sat, 21 Nov 2015 05:52:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.72.227 with HTTP; Sat, 21 Nov 2015 05:52:13 -0800 (PST) In-Reply-To: <56505A29.90408@tngtech.com> References: <1444510765277-3104.post@n4.nabble.com> <9B1D4CB1-CBBA-488E-974E-2064980AF2A5@apache.org> <56505A29.90408@tngtech.com> From: Robert Metzger Date: Sat, 21 Nov 2015 14:52:13 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Processing S3 data with Apache Flink To: "user@flink.apache.org" Content-Type: multipart/alternative; boundary=001a113fb2a4bce76b05250d4d96 --001a113fb2a4bce76b05250d4d96 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, It seems that you've set the "fs.hdfs.hadoopconf" configuration parameter to a file. I think you have to set it the directory containing the configuration. Sorry, I know that's not very intuitive, but in Hadoop the settings for in different files (hdfs|yarn|core)-site.xml. On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf < konstantin.knauf@tngtech.com> wrote: > Hi Ufuk, > > sorry for not getting back to you for so long, and thanks for your > answer. The problem persists unfortunately. Running the job from the IDE > works (with core-site.xml on classpath), running it in local standalone > mode does not. AccessKeyID and SecretAccesKey are not found. > > Attached the jobmanager log on DEBUG level. The core-site.xml is > definitely at the configured location. > > I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 to > run the jar in local mode. Do I have to use the Hadoop 2.x version for > this to work? I have put hadoop-common-2.3.jar into the flink lib folder. > > I don't know if it is relevant (but it seems to be related), when I run > the job from my IDE I get the warning: > > 2015-11-21 12:43:11 WARN NativeCodeLoader:62 - Unable to load > native-hadoop library for your platform... using builtin-java classes > where applicable > > Cheers and thank you, > > Konstantin > > > On 14.10.2015 11:44, Ufuk Celebi wrote: > > > >> On 10 Oct 2015, at 22:59, snntr wrote: > >> > >> Hey everyone, > >> > >> I was having the same problem with S3 and found this thread very usefu= l. > >> Everything works fine now, when I start Flink from my IDE, but when I > run > >> the jar in local mode I keep getting > >> > >> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Acces= s > Key > >> must be specified as the username or password (respectively) of a s3n > URL, > >> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey > >> properties (respectively). > >> > >> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my local > >> machine with the required properties. What am I missing? > >> > >> Any advice is highly appreciated ;) > > > > This looks like a problem with picking up the Hadoop config. Can you > look into the logs to check whether the configuration is picked up? Chang= e > the log settings to DEBUG in log/log4j.properties for this. And can you > provide the complete stack trace? > > > > =E2=80=93 Ufuk > > > > > > -- > Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182 > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring > Gesch=C3=A4ftsf=C3=BChrer: Henrik Klagges, Christoph Stock, Dr. Robert Da= hlke > Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082 > --001a113fb2a4bce76b05250d4d96 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

It seems that you've set the &q= uot;fs.hdfs.hadoopconf" configuration parameter to a file. I think you= have to set it the directory containing the configuration.
Sorry= , I know that's not very intuitive, but in Hadoop the settings for in d= ifferent files (hdfs|yarn|core)-site.xml.


On Sat, Nov 21, 2015 at = 12:48 PM, Konstantin Knauf <konstantin.knauf@tngtech.com>= ; wrote:
Hi Ufuk,

sorry for not getting back to you for so long, and thanks for your
answer. The problem persists unfortunately. Running the job from the IDE works (with core-site.xml on classpath), running it in local standalone
mode does not. AccessKeyID and SecretAccesKey are not found.

Attached the jobmanager log on DEBUG level. The core-site.xml is
definitely at the configured location.

I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1 to
run the jar in local mode. Do I have to use the Hadoop 2.x version for
this to work? I have put hadoop-common-2.3.jar into the flink lib folder.
I don't know if it is relevant (but it seems to be related), when I run=
the job from my IDE I get the warning:

2015-11-21 12:43:11 WARN=C2=A0 NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable

Cheers and thank you,

Konstantin


On 14.10.2015 11:44, Ufuk Celebi wrote:
>
>> On 10 Oct 2015, at 22:59, snntr <konstantin.knauf@tngtech.com> wrote:
>>
>> Hey everyone,
>>
>> I was having the same problem with S3 and found this thread very u= seful.
>> Everything works fine now, when I start Flink from my IDE, but whe= n I run
>> the jar in local mode I keep getting
>>
>> java.lang.IllegalArgumentException: AWS Access Key ID and Secret A= ccess Key
>> must be specified as the username or password (respectively) of a = s3n URL,
>> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessK= ey
>> properties (respectively).
>>
>> I have set fs.hdfs.hadoopconf to point to a core-site.xml on my lo= cal
>> machine with the required properties. What am I missing?
>>
>> Any advice is highly appreciated ;)
>
> This looks like a problem with picking up the Hadoop config. Can you l= ook into the logs to check whether the configuration is picked up? Change t= he log settings to DEBUG in log/log4j.properties for this. And can you prov= ide the complete stack trace?
>
> =E2=80=93 Ufuk
>
>

--
Konstantin Knauf * konstant= in.knauf@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring
Gesch=C3=A4ftsf=C3=BChrer: Henrik Klagges, Christoph Stock, Dr. Robert Dahl= ke
Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082

--001a113fb2a4bce76b05250d4d96--