flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Knauf <konstantin.kn...@tngtech.com>
Subject Re: Processing S3 data with Apache Flink
Date Sat, 21 Nov 2015 14:34:06 GMT
I see, thank you, Robert.

On 21.11.2015 15:28, Robert Metzger wrote:
> Ah, I see. Maybe it would make sense then for you to use the latest
> Hadoop version we are supporting. This way, you get the most recent
> Hadoop S3 file system implementation.
> 
> Note that there might be an issue with starting Flink 0.10.0 for Hadoop
> 2.7.0. We'll fix it with Flink 0.10.1.
> But if everything is working fine ... it might make sense not to change
> it now ("never change a running system").
> 
> 
> On Sat, Nov 21, 2015 at 3:24 PM, Konstantin Knauf
> <konstantin.knauf@tngtech.com <mailto:konstantin.knauf@tngtech.com>> wrote:
> 
>     Hi Robert,
> 
>     I am basically only reading from Kafka and S3 and writing to S3 in this
>     job. So I am using the Hadoop S3 FileSystem classes, but that's it.
> 
>     Cheers,
> 
>     Konstantin
> 
> 
>     On 21.11.2015 15:16, Robert Metzger wrote:
>     > Hi,
>     >
>     > great to hear that its working. I've updated the documentation (for 1.0)
>     > and made the word directory bold ;)
>     >
>     > You should try to match your Hadoop version as closely as possible.
>     > Are you not using HDFS at all? Then it doesn't matter which version of
>     > Flink you are downloading.
>     > When using Hadoop 2.x then I'd recommend at least a Flink version for
>     > Hadoop 2.3.0
>     >
>     >
>     > On Sat, Nov 21, 2015 at 3:13 PM, Konstantin Knauf
>     > <konstantin.knauf@tngtech.com <mailto:konstantin.knauf@tngtech.com>
>     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>>> wrote:
>     >
>     >     Hi Robert,
>     >
>     >     thanks a lot, it's working now. Actually, it also says "directory" in
>     >     the description. So I should have known :/
>     >
>     >     On additional question though. If I use the flink binary for Hadoop
>     >     1.2.1 and run flink in standalone mode, should I use the *-hadoop1
>     >     dependencies even If I am not interacting with HDFS 1.x?
>     >
>     >     Cheers,
>     >
>     >     Konstantin
>     >
>     >     On 21.11.2015 14:52, Robert Metzger wrote:
>     >     > Hi,
>     >     >
>     >     > It seems that you've set the "fs.hdfs.hadoopconf" configuration
>     >     > parameter to a file. I think you have to set it the directory containing
>     >     > the configuration.
>     >     > Sorry, I know that's not very intuitive, but in Hadoop the settings
for
>     >     > in different files (hdfs|yarn|core)-site.xml.
>     >     >
>     >     >
>     >     > On Sat, Nov 21, 2015 at 12:48 PM, Konstantin Knauf
>     >     > <konstantin.knauf@tngtech.com <mailto:konstantin.knauf@tngtech.com>
>     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>>
>     >     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>
>     >     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>>>> wrote:
>     >     >
>     >     >     Hi Ufuk,
>     >     >
>     >     >     sorry for not getting back to you for so long, and thanks for your
>     >     >     answer. The problem persists unfortunately. Running the job from
the IDE
>     >     >     works (with core-site.xml on classpath), running it in local standalone
>     >     >     mode does not. AccessKeyID and SecretAccesKey are not found.
>     >     >
>     >     >     Attached the jobmanager log on DEBUG level. The core-site.xml is
>     >     >     definitely at the configured location.
>     >     >
>     >     >     I am now on version 0.10.0 and using the binaries for Hadoop 1.2.1
to
>     >     >     run the jar in local mode. Do I have to use the Hadoop 2.x version
for
>     >     >     this to work? I have put hadoop-common-2.3.jar into the flink lib
>     >     >     folder.
>     >     >
>     >     >     I don't know if it is relevant (but it seems to be related), when
I run
>     >     >     the job from my IDE I get the warning:
>     >     >
>     >     >     2015-11-21 12:43:11 WARN  NativeCodeLoader:62 - Unable to load
>     >     >     native-hadoop library for your platform... using builtin-java classes
>     >     >     where applicable
>     >     >
>     >     >     Cheers and thank you,
>     >     >
>     >     >     Konstantin
>     >     >
>     >     >
>     >     >     On 14.10.2015 11:44, Ufuk Celebi wrote:
>     >     >     >
>     >     >     >> On 10 Oct 2015, at 22:59, snntr <konstantin.knauf@tngtech.com
<mailto:konstantin.knauf@tngtech.com>
>     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>>
>     >     >     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>
>     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>>>>
>     >     wrote:
>     >     >     >>
>     >     >     >> Hey everyone,
>     >     >     >>
>     >     >     >> I was having the same problem with S3 and found this thread
very
>     >     >     useful.
>     >     >     >> Everything works fine now, when I start Flink from my
IDE, but
>     >     >     when I run
>     >     >     >> the jar in local mode I keep getting
>     >     >     >>
>     >     >     >> java.lang.IllegalArgumentException: AWS Access Key ID
and Secret
>     >     >     Access Key
>     >     >     >> must be specified as the username or password (respectively)
of a
>     >     >     s3n URL,
>     >     >     >> or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey
>     >     >     >> properties (respectively).
>     >     >     >>
>     >     >     >> I have set fs.hdfs.hadoopconf to point to a core-site.xml
on my local
>     >     >     >> machine with the required properties. What am I missing?
>     >     >     >>
>     >     >     >> Any advice is highly appreciated ;)
>     >     >     >
>     >     >     > This looks like a problem with picking up the Hadoop config.
Can
>     >     >     you look into the logs to check whether the configuration is picked
>     >     >     up? Change the log settings to DEBUG in log/log4j.properties for
>     >     >     this. And can you provide the complete stack trace?
>     >     >     >
>     >     >     > – Ufuk
>     >     >     >
>     >     >     >
>     >     >
>     >     >     --
>     >     >     Konstantin Knauf * konstantin.knauf@tngtech.com <mailto:konstantin.knauf@tngtech.com>
>     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>>
>     >     >     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>
>     >     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>>> * +49-174-3413182
>     <tel:%2B49-174-3413182>
>     >     <tel:%2B49-174-3413182>
>     >     >     <tel:%2B49-174-3413182>
>     >     >     TNG Technology Consulting GmbH, Betastr. 13a, 85774
>     Unterföhring
>     >     >     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert
>     >     Dahlke
>     >     >     Sitz: Unterföhring * Amtsgericht München * HRB 135082
>     >     >
>     >     >
>     >
>     >     --
>     >     Konstantin Knauf * konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>
>     >     <mailto:konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com>> * +49-174-3413182
>     <tel:%2B49-174-3413182>
>     >     <tel:%2B49-174-3413182>
>     >     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>     >     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert
>     Dahlke
>     >     Sitz: Unterföhring * Amtsgericht München * HRB 135082
>     >
>     >
> 
>     --
>     Konstantin Knauf * konstantin.knauf@tngtech.com
>     <mailto:konstantin.knauf@tngtech.com> * +49-174-3413182
>     <tel:%2B49-174-3413182>
>     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>     Sitz: Unterföhring * Amtsgericht München * HRB 135082
> 
> 

-- 
Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082

Mime
View raw message