Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CDDC81831F for ; Sun, 8 Nov 2015 18:06:33 +0000 (UTC) Received: (qmail 65286 invoked by uid 500); 8 Nov 2015 18:06:33 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 65208 invoked by uid 500); 8 Nov 2015 18:06:33 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 65198 invoked by uid 99); 8 Nov 2015 18:06:33 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Nov 2015 18:06:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 300321A07F2 for ; Sun, 8 Nov 2015 18:06:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.001 X-Spam-Level: *** X-Spam-Status: No, score=3.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=simplydevelop_de.20150623.gappssmtp.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id LEop6FmPZWT7 for ; Sun, 8 Nov 2015 18:06:17 +0000 (UTC) Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id E768A24B3C for ; Sun, 8 Nov 2015 18:06:16 +0000 (UTC) Received: by wmnn186 with SMTP id n186so81043255wmn.1 for ; Sun, 08 Nov 2015 10:06:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=simplydevelop_de.20150623.gappssmtp.com; s=20150623; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=oQAKq8FPTa0Npg7ezQShnhmLY2/kpa7Hfgy6upxNl9E=; b=yXolA3wfn1JyIrmsU9jxnq2duI3VGrwRMoHl9aQs71uHGPhDtmB6VKFtqD59doU3F2 hitMOSsnZa+dgkfbenzVeiupDBjd7XObLKwKIzamdkJnbDyCAdLiJC0Ul5Lj7X+hJsw7 pKxVHBRMfjLD44oP8KJxfK8+BOYqzwI0ewEdBqdxbMnT96Ela7Bd/CytYoXLqVHdCdet nB9vH3tDFEBcQEghoW4GTsQxrB/rW0mN83saRX+L0uP2tK+qVqt2AuT3uAVYTo92THPB CSuLTE3E/vD6ePcC3B8c9MPme8hmTXho5KSr9ze6NUCAZp9N1hm8a7NmqCSkuG0ab0vr hRuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=oQAKq8FPTa0Npg7ezQShnhmLY2/kpa7Hfgy6upxNl9E=; b=CxRQEKcc0MqyCFJ+htEfC49qVTCCkZhHoQuLalbdDLkeRKhOJ6O/s8yRMJB+lXj34y sax+Tn/JaNT+6TwSiXn5kCjTlF2jVVuJEWJ/iG7X8P0gIPvfGDMRAFpd4iafSs97FOeh AJPp4Vf52H0m5rzntxqG5Kj7jctW78J9SR/zfwl290JzLVoxGHZ7XEkYojIpS8iBhKsh rZLsRB7ukq8RwV8LlQNJfGYLl4CIyAlaE/tIZQuK20Ilqy43B8uLUoGyRYJgfhjLmdXS XyDJ9pMJPTsSusM44+7ZB+5cXALofkZeaS4A3mpqQ9g934pKeRpReOQFbdYVb23mHtKd 5elQ== X-Gm-Message-State: ALoCoQnfMrCDhVaWybl4/+U1bouqdJLg2fZIXM4gnw6Q8LcDMT8jqTxQg/OKglB1Pft/pX59opfE X-Received: by 10.28.174.195 with SMTP id x186mr22671537wme.87.1447005976576; Sun, 08 Nov 2015 10:06:16 -0800 (PST) Received: from thomass-mbp.fritz.box ([37.9.188.6]) by smtp.gmail.com with ESMTPSA id q1sm11038728wjy.31.2015.11.08.10.06.15 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 08 Nov 2015 10:06:16 -0800 (PST) From: =?utf-8?Q?Thomas_G=C3=B6tzinger?= Content-Type: multipart/alternative; boundary="Apple-Mail=_A4950478-4FBA-4877-ABE6-9F676E9BF656" Message-Id: <152E0794-CFF0-4285-BD8A-01077628F8C1@simplydevelop.de> Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Subject: Re: Flink on EC" Date: Sun, 8 Nov 2015 19:06:14 +0100 References: To: user@flink.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3096.5) --Apple-Mail=_A4950478-4FBA-4877-ABE6-9F676E9BF656 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 HI Fabian, thanks for reply. I use a karamel receipt to install flink on = ec2.Currently I am using flink-0.9.1-bin-hadoop24.tgz = . In that file the NativeS3FileSystem is included. First I=E2=80=99ve = tried it with the standard karamel receipt on github = hopshadoop/flink-chef but = it=E2=80=99s on Version 0.9.0 and the S3NFileSystem is not included. So I forked the github project by goetzingert/flink-chef Although the class file is include the application throws a = ClassNotFoundException for the class above. In my Project I add the conf/core-site.xml fs.s3n.impl org.apache.hadoop.fs.s3native.NativeS3FileSystem fs.s3n.awsAccessKeyId =E2=80=A6. fs.s3n.awsSecretAccessKey ... =E2=80=94=20 I also tried to use the programmatic configuration=20 XMLConfiguration config =3D new = XMLConfiguration(configPath); env =3D ExecutionEnvironment.getExecutionEnvironment(); Configuration configuration =3D = GlobalConfiguration.getConfiguration(); configuration.setString("fs.s3.impl", = "org.apache.hadoop.fs.s3native.NativeS3FileSystem"); configuration.setString("fs.s3n.awsAccessKeyId", = =E2=80=9C.."); = configuration.setString("fs.s3n.awsSecretAccessKey=E2=80=9D,=E2=80=9D../")= ; = configuration.setString("fs.hdfs.hdfssite",Template.class.getResource("/co= nf/core-site.xml").toString()); GlobalConfiguration.includeConfiguration(configuration); Any Idea why the class is not included in classpath? Is there another = script to setup flink on ec2 cluster? When will flink 0.10 be released?=20 Regards =20 Thomas G=C3=B6tzinger Freiberuflicher Informatiker =20 Glockenstra=C3=9Fe 2a D-66882 H=C3=BCtschenhausen OT Spesbach Mobil: +49 (0)176 82180714 Privat: +49 (0) 6371 954050 mailto:mail@simplydevelop.de epost: thomas.goetzinger@epost.de > On 29.10.2015, at 09:47, Fabian Hueske wrote: >=20 > Hi Thomas, >=20 > until recently, Flink provided an own implementation of a S3FileSystem = which wasn't fully tested and buggy. > We removed that implementation and are using now (in 0.10-SNAPSHOT) = Hadoop's S3 implementation by default. >=20 > If you want to continue using 0.9.1 you can configure Flink to use = Hadoop's implementation. See this answer on StackOverflow and the linked = email thread [1]. > If you switch to the 0.10-SNAPSHOT version (which will be released in = a few days as 0.10.0), things become a bit easier and Hadoop's = implementation is used by default. The documentation shows how to = configure your access keys [2] >=20 > Please don't hesitate to ask if something is unclear or not working. >=20 > Best, Fabian >=20 > [1] = http://stackoverflow.com/questions/32959790/run-apache-flink-with-amazon-s= 3 = > [2] = https://ci.apache.org/projects/flink/flink-docs-master/apis/example_connec= tors.html = >=20 > 2015-10-29 9:35 GMT+01:00 Thomas G=C3=B6tzinger >: > Hello Flink Team, >=20 > We at IESE Fraunhofer are evaluating Flink for a project and I'm a bit = frustrated in the moment.=20 >=20 > I've wrote a few testcases with the flink API and want to deploy them = to an Flink EC2 Cluster. I setup the cluster using the=20 > karamel receipt which was adressed in the following video=20 >=20 > = https://www.google.de/url?sa=3Dt&rct=3Dj&q=3D&esrc=3Ds&source=3Dvideo&cd=3D= 1&cad=3Drja&uact=3D8&ved=3D0CDIQtwIwAGoVChMIy86Tq6rQyAIVR70UCh0IRwuJ&url=3D= http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dm_SkhyMV0to&usg=3DAFQjCNGKUzFv5= 21yg-OTy-1XqS2-rbZKug&bvm=3Dbv.105454873,d.bGg = >=20 > The setup works fine and the hello-flink app could be run. But = afterwards I want to copy some data from s3 bucket to the local ec2 hdfs = cluster.=20 >=20 > The hadoop fs -ls s3n.... works as well as cat,... > But if I want to copy the data with distcp the command freezes, and = does not respond until a timeout. >=20 > After trying a few things I gave up and start another solution. I want = to access the s3 Bucket directly with flink and import it using a small = flink programm which just reads s3 and writes to local hadoop. This = works fine locally, but on cluster the S3NFileSystem class is missing = (ClassNotFound Exception) althoug it is included in the jar file of the = installation.=20 >=20 >=20 > I forked the chef receipt and updated to flink 0.9.1 but the same = issue. >=20 > Is there another simple script to install flink with hadoop on an ec2 = cluster and working s3n filesystem? >=20 >=20 >=20 >=20 > Freelancer=20 >=20 > on Behalf of Fraunhofer IESE Kaiserslautern >=20 >=20 > --=20 > Viele Gr=C3=BC=C3=9Fe >=20 > =20 > Thomas G=C3=B6tzinger >=20 > Freiberuflicher Informatiker >=20 > =20 > Glockenstra=C3=9Fe 2a >=20 > D-66882 H=C3=BCtschenhausen OT Spesbach >=20 > Mobil: +49 (0)176 82180714 >=20 > Homezone: +49 (0) 6371 735083 >=20 > Privat: +49 (0) 6371 954050 >=20 > mailto:mail@simplydevelop.de > epost: thomas.goetzinger@epost.de --Apple-Mail=_A4950478-4FBA-4877-ABE6-9F676E9BF656 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 HI Fabian,

thanks for reply. I use a karamel receipt to install flink on = ec2.Currently I am using flink-0.9.1-bin-hadoop24.tgz.

 In that file the = NativeS3FileSystem is included. First I=E2=80=99ve tried it with the = standard karamel receipt on github hopshadoop/flink-chef but = it=E2=80=99s on Version 0.9.0 and the S3NFileSystem is not = included.
So I forked the github project by = goetzingert/flink-chef
Although the class file is = include the application throws a ClassNotFoundException for the class = above.
In my Project I add the = conf/core-site.xml

  <property>
    <name>fs.s3n.impl</name>
    = <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
  </property>
  <property>
    = <name>fs.s3n.awsAccessKeyId</name>
    = <value>=E2=80=A6.</value>
  </property>
  <property>
    = <name>fs.s3n.awsSecretAccessKey</name>
    = <value>...</value>
  </property>

=E2=80=94 
I also tried to use the programmatic = configuration 

= XMLConfiguration config =3D new XMLConfiguration(configPath);

= env =3D = ExecutionEnvironment.getExecutionEnvironment();
= Configuration configuration =3D = GlobalConfiguration.getConfiguration();
= configuration.setString("fs.s3.impl", = "org.apache.hadoop.fs.s3native.NativeS3FileSystem");
= configuration.setString("fs.s3n.awsAccessKeyId", =E2=80=9C..");
= configuration.setString("fs.s3n.awsSecretAccessKey=E2=80=9D,=E2=80=9D../");
= configuration.setString("fs.hdfs.hdfssite",Template.class.getResource("/conf/core-site.xml").toString());
= GlobalConfiguration.includeConfiguration(configuration);


Any Idea why the class is not included in classpath? Is there another = script to setup flink on ec2 cluster?

When will flink 0.10 be = released? 


Regards

 

Thomas = G=C3=B6tzinger

Freiberuflicher = Informatiker

 

Glockenstra=C3= =9Fe 2a

D-66882 H=C3=BCtschenhausen OT = Spesbach

Mobil: +49 (0)176 = 82180714

Privat:= +49 (0) 6371 954050

mailto:mail@simplydevelop.de

epost: thomas.goetzinger@epost.de




On 29.10.2015, at 09:47, Fabian Hueske <fhueske@gmail.com> = wrote:

Hi Thomas,

until recently, Flink provided an own = implementation of a S3FileSystem which wasn't fully tested and buggy.
We removed that implementation and are = using now (in 0.10-SNAPSHOT) Hadoop's S3 implementation by default.

If you want to continue = using 0.9.1 you can configure Flink to use Hadoop's implementation. See = this answer on StackOverflow and the linked email thread [1].
If you switch to the 0.10-SNAPSHOT = version (which will be released in a few days as 0.10.0), things become = a bit easier and Hadoop's implementation is used by default. The = documentation shows how to configure your access keys [2]

Please don't hesitate to ask if something is unclear or not = working.

Best, = Fabian

[1] http://stackoverflow.com/questions/32959790/run-apache-flink-wi= th-amazon-s3
[2] https://ci.apache.org/projects/flink/flink-docs-master/apis/exa= mple_connectors.html

2015-10-29= 9:35 GMT+01:00 Thomas G=C3=B6tzinger <mail@simplydevelop.de>:
Hello Flink = Team,

We= at IESE Fraunhofer are evaluating Flink for a project and I'm = a bit frustrated in the moment. 

I've wrote a few testcases with the flink API and = want to deploy them to an Flink EC2 Cluster. I setup the = cluster using the 
karamel receipt which was adressed in the following = video 

https://www.google.de/url?sa=3Dt&rct=3Dj&q=3D&esrc=3D= s&source=3Dvideo&cd=3D1&cad=3Drja&uact=3D8&ved=3D0CDIQ= twIwAGoVChMIy86Tq6rQyAIVR70UCh0IRwuJ&url=3Dhttp%3A%2F%2Fwww.youtube.co= m%2Fwatch%3Fv%3Dm_SkhyMV0to&usg=3DAFQjCNGKUzFv521yg-OTy-1XqS2-rbZKug&a= mp;bvm=3Dbv.105454873,d.bGg

The setup works fine and the hello-flink app could be = run. But afterwards I want to copy some data from s3 bucket to the local = ec2 hdfs cluster. 

The hadoop fs -ls = s3n.... works as well as cat,...
But if I want to copy = the data with distcp the command freezes, and does not respond until a = timeout.

After trying a few = things I gave up and start another solution. I want to access the s3 = Bucket directly with flink and import it using a = small flink programm which just reads s3 and writes to local = hadoop. This works fine locally, but on cluster the S3NFileSystem class = is missing (ClassNotFound Exception) althoug it is included in the jar = file of the installation. 


I = forked the chef receipt and updated to flink 0.9.1 but the = same issue.

Is = there another simple script to install flink with hadoop on an = ec2 cluster and working s3n filesystem?



Freelancer 

on Behalf of Fraunhofer IESE Kaiserslautern


--

Viele Gr=C3=BC=C3=9Fe

 

Thomas G=C3=B6tzinger

Freiberuflicher Informatiker

 

Glockenstra=C3=9Fe 2a

D-66882 H=C3=BCtschenhausen OT Spesbach

Mobil: +49 (0)176 82180714

Homezone: +49 (0) 6371 735083

Privat: +49 (0) 6371 954050

mailto:mail@simplydevelop.de

epost: thomas.goetzinger@epost.de



= --Apple-Mail=_A4950478-4FBA-4877-ABE6-9F676E9BF656--