flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KOSTIANTYN Kudriavtsev <kudryavtsev.konstan...@gmail.com>
Subject Re: Flink on EC"
Date Thu, 29 Oct 2015 13:47:54 GMT
Hi Thomas,

Try to switch to Emr amo 3.5 and register hadoop's s3 FileSystem instead of
the one packed with flink

*Sent from my ZenFone
On Oct 29, 2015 4:36 AM, "Thomas Götzinger" <mail@simplydevelop.de> wrote:

> Hello Flink Team,
>
> We at IESE Fraunhofer are evaluating Flink for a project and I'm a bit
> frustrated in the moment.
>
> I've wrote a few testcases with the flink API and want to deploy them to
> an Flink EC2 Cluster. I setup the cluster using the
> karamel receipt which was adressed in the following video
>
>
> https://www.google.de/url?sa=t&rct=j&q=&esrc=s&source=video&cd=1&cad=rja&uact=8&ved=0CDIQtwIwAGoVChMIy86Tq6rQyAIVR70UCh0IRwuJ&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dm_SkhyMV0to&usg=AFQjCNGKUzFv521yg-OTy-1XqS2-rbZKug&bvm=bv.105454873,d.bGg
>
> The setup works fine and the hello-flink app could be run. But afterwards
> I want to copy some data from s3 bucket to the local ec2 hdfs cluster.
>
> The hadoop fs -ls s3n.... works as well as cat,...
> But if I want to copy the data with distcp the command freezes, and does
> not respond until a timeout.
>
> After trying a few things I gave up and start another solution. I want to
> access the s3 Bucket directly with flink and import it using a
> small flink programm which just reads s3 and writes to local hadoop. This
> works fine locally, but on cluster the S3NFileSystem class is missing
> (ClassNotFound Exception) althoug it is included in the jar file of the
> installation.
>
>
> I forked the chef receipt and updated to flink 0.9.1 but the same issue.
>
> Is there another simple script to install flink with hadoop on an ec2
> cluster and working s3n filesystem?
>
>
>
> Freelancer
>
> on Behalf of Fraunhofer IESE Kaiserslautern
>
> --
>
> Viele Grüße
>
>
>
> Thomas Götzinger
>
> Freiberuflicher Informatiker
>
>
>
> Glockenstraße 2a
>
> D-66882 Hütschenhausen OT Spesbach
>
> Mobil: +49 (0)176 82180714
>
> Homezone: +49 (0) 6371 735083
>
> Privat: +49 (0) 6371 954050
>
> mailto:mail@simplydevelop.de <thomas.goetzinger@kajukin.de>
>
> epost: thomas.goetzinger@epost.de
>

Mime
View raw message