Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 314A317C83 for ; Thu, 4 Jun 2015 15:38:50 +0000 (UTC) Received: (qmail 93159 invoked by uid 500); 4 Jun 2015 15:38:48 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 93097 invoked by uid 500); 4 Jun 2015 15:38:48 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 93087 invoked by uid 99); 4 Jun 2015 15:38:48 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Jun 2015 15:38:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 86844CB59F for ; Thu, 4 Jun 2015 15:38:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.151 X-Spam-Level: ***** X-Spam-Status: No, score=5.151 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, KAM_BADIPHTTP=2, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=googlemail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id egwrpKetjIt6 for ; Thu, 4 Jun 2015 15:38:35 +0000 (UTC) Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 097B6453B6 for ; Thu, 4 Jun 2015 15:38:35 +0000 (UTC) Received: by wibdt2 with SMTP id dt2so53802261wib.1 for ; Thu, 04 Jun 2015 08:38:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=9DrxhNUTG4EznZxakounlsaWpXfRg6bMSK97WUvGL/Y=; b=jdgIA+z6w+oDt5qUB6c1FPL/8I6P+2GfJsSOJpZFNZROI2wHw2seDmyzG5IO++RQbc vYtdhwHkEXadgdeexlh3Vrbh4R3zWcuhdbsnZ8linKV5iG3wGlUHRF4wKhh+az6wc8ZA Dz8dh+u3Ts927S1TOoFJsOyFFbPbsDI9/R3W+a6ji4jsHfigo5vN44Hz+13UBshJlY9L uR8pyowWuxORe2Kb40LCrrozo/+breibpp6zH4+21YpboJRVNKp2mM0BbPdG7udNzAqg b/f2CeYYuABn2lVlL0BHxb3uY4vL+tAGoyUZ612FIfnP3ztS0xh51c4BFqvwfNtgRaBL Ab4g== MIME-Version: 1.0 X-Received: by 10.194.205.5 with SMTP id lc5mr74320355wjc.74.1433432314178; Thu, 04 Jun 2015 08:38:34 -0700 (PDT) Received: by 10.180.148.104 with HTTP; Thu, 4 Jun 2015 08:38:34 -0700 (PDT) In-Reply-To: References: Date: Thu, 4 Jun 2015 17:38:34 +0200 Message-ID: Subject: Re: flink k-means on hadoop cluster From: =?UTF-8?B?UGEgUsO2?= To: user@flink.apache.org Content-Type: multipart/alternative; boundary=047d7b874996e5744d0517b2f754 --047d7b874996e5744d0517b2f754 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable sorry, i see my yarn end before i can run my app, i must set the write access for yarn, maybe this solve my problem. 2015-06-04 17:33 GMT+02:00 Pa R=C3=B6 : > i start the yarn-session.sh with sudo > and than the flink run command with sudo, > i get the following exception: > > cloudera@quickstart bin]$ sudo ./flink run > /home/cloudera/Desktop/ma-flink.jar > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info. > org.apache.flink.client.program.ProgramInvocationException: Failed to > resolve JobManager > at org.apache.flink.client.program.Client.run(Client.java:378) > at org.apache.flink.client.program.Client.run(Client.java:355) > at org.apache.flink.client.program.Client.run(Client.java:348) > at > org.apache.flink.client.program.ContextEnvironment.execute(ContextEnviron= ment.java:63) > at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:70) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedPr= ogram.java:437) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForE= xecution(PackagedProgram.java:353) > at org.apache.flink.client.program.Client.run(Client.java:315) > at > org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584) > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) > at > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:880) > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922) > Caused by: java.io.IOException: JobManager at akka.tcp:// > flink@127.0.0.1:6123/user/jobmanager not reachable. Please make sure that > the JobManager is running and its port is reachable. > at > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteRefere= nce(JobManager.scala:1198) > at > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteRefere= nce(JobManager.scala:1222) > at > org.apache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteRefere= nce(JobManager.scala:1240) > at > org.apache.flink.runtime.jobmanager.JobManager.getJobManagerRemoteReferen= ce(JobManager.scala) > at org.apache.flink.client.program.Client.run(Client.java:375) > ... 15 more > Caused by: akka.actor.ActorNotFound: Actor not found for: > ActorSelection[Anchor(akka.tcp://flink@127.0.0.1:6123/), > Path(/user/jobmanager)] > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal= a:65) > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal= a:63) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(Batchi= ngExecutor.scala:67) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Batching= Executor.scala:82) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecuto= r.scala:59) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecuto= r.scala:59) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) > at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58= ) > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExec= ute(Future.scala:74) > at > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Futur= e.scala:73) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:24= 8) > at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) > at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) > at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) > at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) > at > akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteA= ctorRefProvider.scala:87) > at akka.remote.EndpointWriter.postStop(Endpoint.scala:561) > at akka.actor.Actor$class.aroundPostStop(Actor.scala:475) > at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) > at > akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$= finishTerminate(FaultHandling.scala:210) > at > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) > at akka.actor.ActorCell.terminate(ActorCell.scala:369) > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279) > at akka.dispatch.Mailbox.run(Mailbox.scala:220) > at akka.dispatch.Mailbox.exec(Mailbox.scala:231) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:26= 0) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.jav= a:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.j= ava:107) > > the FlinkMain.java: 70 is: > > env.execute("KMeans Flink"); > > > 2015-06-04 17:17 GMT+02:00 Pa R=C3=B6 : > >> i try this: >> >> [cloudera@quickstart bin]$ sudo su yarn >> bash-4.1$ hadoop fs -chmod 777 /user/cloudera/outputs >> chmod: changing permissions of '/user/cloudera/outputs': Permission >> denied. user=3Dyarn is not the owner of inode=3Doutputs >> bash-4.1$ hadoop fs -chmod 777 /user/cloudera/inputs >> chmod: changing permissions of '/user/cloudera/inputs': Permission >> denied. user=3Dyarn is not the owner of inode=3Dinputs >> bash-4.1$ exit >> exit >> [cloudera@quickstart bin]$ sudo ./flink run >> /home/cloudera/Desktop/ma-flink.jar >> log4j:WARN No appenders could be found for logger >> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). >> log4j:WARN Please initialize the log4j system properly. >> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for >> more info. >> Found YARN properties file >> /home/cloudera/Desktop/flink-0.9-SNAPSHOT/bin/../conf/.yarn-properties >> Using JobManager address from YARN properties quickstart.cloudera/ >> 127.0.0.1:53874 >> org.apache.flink.client.program.ProgramInvocationException: The program >> execution failed: Failed to submit job 2f46ef5dff4ecf5552b3477ed1c6f4b9 >> (KMeans Flink) >> at org.apache.flink.client.program.Client.run(Client.java:412) >> at org.apache.flink.client.program.Client.run(Client.java:355) >> at org.apache.flink.client.program.Client.run(Client.java:348) >> at >> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnviro= nment.java:63) >> at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:70) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav= a:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor= Impl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedP= rogram.java:437) >> at >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeFor= Execution(PackagedProgram.java:353) >> at org.apache.flink.client.program.Client.run(Client.java:315) >> at >> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584) >> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) >> at >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:880= ) >> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922) >> Caused by: org.apache.flink.runtime.client.JobExecutionException: Failed >> to submit job 2f46ef5dff4ecf5552b3477ed1c6f4b9 (KMeans Flink) >> at org.apache.flink.runtime.jobmanager.JobManager.org >> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:= 595) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMe= ssages$1.applyOrElse(JobManager.scala:192) >> at >> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPart= ialFunction.scala:33) >> at >> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunct= ion.scala:33) >> at >> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunct= ion.scala:25) >> at >> org.apache.flink.yarn.ApplicationMasterActor$$anonfun$receiveYarnMessage= s$1.applyOrElse(ApplicationMasterActor.scala:99) >> at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) >> at >> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages= .scala:36) >> at >> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages= .scala:29) >> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118= ) >> at >> org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMe= ssages.scala:29) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >> at >> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.= scala:94) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) >> at akka.dispatch.Mailbox.run(Mailbox.scala:221) >> at akka.dispatch.Mailbox.exec(Mailbox.scala:231) >> at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.ja= va:1339) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.= java:107) >> Caused by: org.apache.flink.runtime.JobException: Creating the input >> splits caused an error: File /user/cloudera/inputs does not exist or the >> user running Flink ('yarn') has insufficient permissions to access it. >> at >> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(Execut= ionJobVertex.java:162) >> at >> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(Ex= ecutionGraph.java:471) >> at org.apache.flink.runtime.jobmanager.JobManager.org >> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:= 535) >> ... 21 more >> Caused by: java.io.FileNotFoundException: File /user/cloudera/inputs doe= s >> not exist or the user running Flink ('yarn') has insufficient permission= s >> to access it. >> at >> org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSy= stem.java:106) >> at >> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInp= utFormat.java:390) >> at >> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInp= utFormat.java:51) >> at >> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(Execut= ionJobVertex.java:146) >> ... 23 more >> >> >> 2015-06-04 17:15 GMT+02:00 Robert Metzger : >> >>> As the output of the "hadoop" tool indicates, it expects two arguments, >>> you only passed one (777). >>> The second argument it is expecting is the path to the file you want to >>> change. >>> >>> In your case, it is: >>> hadoop fs -chmod 777 /user/cloudera/outputs >>> >>> >>> The reason why >>> hadoop fs -chmod 777 * >>> does not work is the following: the * is evaluated by your local bash >>> and expanded to the files which are present in your current, local >>> directory. The bash expansion is not able to expand to the files in HDF= S. >>> >>> >>> On Thu, Jun 4, 2015 at 5:08 PM, Pa R=C3=B6 >>> wrote: >>> >>>> [cloudera@quickstart bin]$ sudo su yarn >>>> bash-4.1$ hadoop fs -chmod 777 >>>> -chmod: Not enough arguments: expected 2 but got 1 >>>> Usage: hadoop fs [generic options] -chmod [-R] >>> OCTALMODE> PATH... >>>> bash-4.1$ >>>> >>>> you understand? >>>> >>>> 2015-06-04 17:04 GMT+02:00 Robert Metzger : >>>> >>>>> It looks like the user "yarn" which is running Flink doesn't have >>>>> permission to access the files. >>>>> >>>>> Can you do "sudo su yarn" to become the "yarn" user. Then, you can do >>>>> "hadoop fs -chmod 777" to make the files accessible for everyone. >>>>> >>>>> >>>>> On Thu, Jun 4, 2015 at 4:59 PM, Pa R=C3=B6 >>>>> wrote: >>>>> >>>>>> okay, it's work, i get a exception: >>>>>> >>>>>> [cloudera@quickstart Desktop]$ cd flink-0.9-SNAPSHOT/bin/ >>>>>> [cloudera@quickstart bin]$ flink run >>>>>> /home/cloudera/Desktop/ma-flink.jar >>>>>> bash: flink: command not found >>>>>> [cloudera@quickstart bin]$ ./flink run >>>>>> /home/cloudera/Desktop/ma-flink.jar >>>>>> log4j:WARN No appenders could be found for logger >>>>>> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). >>>>>> log4j:WARN Please initialize the log4j system properly. >>>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>>>>> for more info. >>>>>> Found YARN properties file >>>>>> /home/cloudera/Desktop/flink-0.9-SNAPSHOT/bin/../conf/.yarn-properti= es >>>>>> Using JobManager address from YARN properties quickstart.cloudera/ >>>>>> 127.0.0.1:53874 >>>>>> java.io.IOException: Mkdirs failed to create /user/cloudera/outputs >>>>>> at >>>>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.ja= va:438) >>>>>> at >>>>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.ja= va:424) >>>>>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905) >>>>>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886) >>>>>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783) >>>>>> at >>>>>> mgm.tp.bigdata.ma_commons.commons.Seeding.randomSeeding(Seeding.java= :21) >>>>>> at >>>>>> mgm.tp.bigdata.ma_flink.FlinkMain.getCentroidDataSet(FlinkMain.java:= 178) >>>>>> at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:47) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl= .java:57) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> at >>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(Packa= gedProgram.java:437) >>>>>> at >>>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveMod= eForExecution(PackagedProgram.java:353) >>>>>> at org.apache.flink.client.program.Client.run(Client.java:315) >>>>>> at >>>>>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:= 584) >>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) >>>>>> at >>>>>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java= :880) >>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922= ) >>>>>> org.apache.flink.client.program.ProgramInvocationException: The >>>>>> program execution failed: Failed to submit job >>>>>> 934743a5c49c6d5e31c9e8201452e36d (KMeans Flink) >>>>>> at org.apache.flink.client.program.Client.run(Client.java:412) >>>>>> at org.apache.flink.client.program.Client.run(Client.java:355) >>>>>> at org.apache.flink.client.program.Client.run(Client.java:348) >>>>>> at >>>>>> org.apache.flink.client.program.ContextEnvironment.execute(ContextEn= vironment.java:63) >>>>>> at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:70) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> at >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl= .java:57) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> at >>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(Packa= gedProgram.java:437) >>>>>> at >>>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveMod= eForExecution(PackagedProgram.java:353) >>>>>> at org.apache.flink.client.program.Client.run(Client.java:315) >>>>>> at >>>>>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:= 584) >>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) >>>>>> at >>>>>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java= :880) >>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922= ) >>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: >>>>>> Failed to submit job 934743a5c49c6d5e31c9e8201452e36d (KMeans Flink) >>>>>> at org.apache.flink.runtime.jobmanager.JobManager.org >>>>>> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.sc= ala:595) >>>>>> at >>>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithL= ogMessages$1.applyOrElse(JobManager.scala:192) >>>>>> at >>>>>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(Abstract= PartialFunction.scala:33) >>>>>> at >>>>>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialF= unction.scala:33) >>>>>> at >>>>>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialF= unction.scala:25) >>>>>> at >>>>>> org.apache.flink.yarn.ApplicationMasterActor$$anonfun$receiveYarnMes= sages$1.applyOrElse(ApplicationMasterActor.scala:99) >>>>>> at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) >>>>>> at >>>>>> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMess= ages.scala:36) >>>>>> at >>>>>> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMess= ages.scala:29) >>>>>> at >>>>>> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) >>>>>> at >>>>>> org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorL= ogMessages.scala:29) >>>>>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>>>>> at >>>>>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobMana= ger.scala:94) >>>>>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>>>>> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>>>>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) >>>>>> at akka.dispatch.Mailbox.run(Mailbox.scala:221) >>>>>> at akka.dispatch.Mailbox.exec(Mailbox.scala:231) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPoo= l.java:1339) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1= 979) >>>>>> at >>>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThr= ead.java:107) >>>>>> Caused by: org.apache.flink.runtime.JobException: Creating the input >>>>>> splits caused an error: File /user/cloudera/outputs/seed-1 does not = exist >>>>>> or the user running Flink ('yarn') has insufficient permissions to a= ccess >>>>>> it. >>>>>> at >>>>>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(Ex= ecutionJobVertex.java:162) >>>>>> at >>>>>> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGrap= h(ExecutionGraph.java:471) >>>>>> at org.apache.flink.runtime.jobmanager.JobManager.org >>>>>> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.sc= ala:535) >>>>>> ... 21 more >>>>>> Caused by: java.io.FileNotFoundException: File >>>>>> /user/cloudera/outputs/seed-1 does not exist or the user running Fli= nk >>>>>> ('yarn') has insufficient permissions to access it. >>>>>> at >>>>>> org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFi= leSystem.java:106) >>>>>> at >>>>>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(Fil= eInputFormat.java:390) >>>>>> at >>>>>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(Fil= eInputFormat.java:51) >>>>>> at >>>>>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(Ex= ecutionJobVertex.java:146) >>>>>> ... 23 more >>>>>> >>>>>> >>>>>> how i must set the files in the hdfs? >>>>>> quickstart.cloudera:50075/home/cloudera/output? >>>>>> >>>>>> 2015-06-04 16:51 GMT+02:00 Robert Metzger : >>>>>> >>>>>>> Once you've started the YARN session, you can submit a Flink job >>>>>>> with "./bin/flink run ". >>>>>>> >>>>>>> The jar file of your job doesn't need to be in HDFS. It has to be i= n >>>>>>> the local file system and flink will send it to all machines. >>>>>>> >>>>>>> On Thu, Jun 4, 2015 at 4:48 PM, Pa R=C3=B6 < >>>>>>> paul.roewer1990@googlemail.com> wrote: >>>>>>> >>>>>>>> okay, now it run on my hadoop. >>>>>>>> how i can start my flink job? and where must the jar file save, at >>>>>>>> hdfs or as local file? >>>>>>>> >>>>>>>> 2015-06-04 16:31 GMT+02:00 Robert Metzger : >>>>>>>> >>>>>>>>> Yes, you have to run these commands in the command line of the >>>>>>>>> Cloudera VM. >>>>>>>>> >>>>>>>>> On Thu, Jun 4, 2015 at 4:28 PM, Pa R=C3=B6 < >>>>>>>>> paul.roewer1990@googlemail.com> wrote: >>>>>>>>> >>>>>>>>>> you mean run this command on terminal/shell and not define a hue >>>>>>>>>> job? >>>>>>>>>> >>>>>>>>>> 2015-06-04 16:25 GMT+02:00 Robert Metzger : >>>>>>>>>> >>>>>>>>>>> It should be certainly possible to run Flink on a cloudera live >>>>>>>>>>> VM >>>>>>>>>>> >>>>>>>>>>> I think these are the commands you need to execute: >>>>>>>>>>> >>>>>>>>>>> wget >>>>>>>>>>> http://stratosphere-bin.s3-website-us-east-1.amazonaws.com/flin= k-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>>>>>> tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>>>>>> cd flink-0.9-SNAPSHOT/ >>>>>>>>>>> *export HADOOP_CONF_DIR=3D/usr/lib/hadoop/etc/hadoop/* >>>>>>>>>>> ./bin/yarn-session.sh -n 1 -jm 1024 -tm 1024 >>>>>>>>>>> >>>>>>>>>>> If that is not working for you, please post the exact error >>>>>>>>>>> message you are getting and I can help you to get it to run. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 4, 2015 at 4:18 PM, Pa R=C3=B6 < >>>>>>>>>>> paul.roewer1990@googlemail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> hi robert, >>>>>>>>>>>> >>>>>>>>>>>> i think the problem is the hue api, >>>>>>>>>>>> i had the same problem with spark submit script, >>>>>>>>>>>> but on the new hue release, they have a spark submit api. >>>>>>>>>>>> >>>>>>>>>>>> i asked the group for the same problem with spark, no reply. >>>>>>>>>>>> >>>>>>>>>>>> i want test my app on local cluster, before i run it on the bi= g >>>>>>>>>>>> cluster, >>>>>>>>>>>> for that i use cloudera live. maybe it give an other way to >>>>>>>>>>>> test flink on a local cluster vm? >>>>>>>>>>>> >>>>>>>>>>>> 2015-06-04 16:12 GMT+02:00 Robert Metzger >>>>>>>>>>>> : >>>>>>>>>>>> >>>>>>>>>>>>> Hi Paul, >>>>>>>>>>>>> >>>>>>>>>>>>> why did running Flink from the regular scripts not work for >>>>>>>>>>>>> you? >>>>>>>>>>>>> >>>>>>>>>>>>> I'm not an expert on Hue, I would recommend asking in the Hue >>>>>>>>>>>>> user forum / mailing list: >>>>>>>>>>>>> https://groups.google.com/a/cloudera.org/forum/#!forum/hue-us= er >>>>>>>>>>>>> . >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jun 4, 2015 at 4:09 PM, Pa R=C3=B6 < >>>>>>>>>>>>> paul.roewer1990@googlemail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> thanks, >>>>>>>>>>>>>> now i want run my app on cloudera live vm single node, >>>>>>>>>>>>>> how i can define my flink job with hue? >>>>>>>>>>>>>> i try to run the flink script in the hdfs, it's not work. >>>>>>>>>>>>>> >>>>>>>>>>>>>> best regards, >>>>>>>>>>>>>> paul >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2015-06-02 14:50 GMT+02:00 Robert Metzger < >>>>>>>>>>>>>> rmetzger@apache.org>: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I would recommend using HDFS. >>>>>>>>>>>>>>> For that, you need to specify the paths like this: >>>>>>>>>>>>>>> hdfs:///path/to/data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Jun 2, 2015 at 2:48 PM, Pa R=C3=B6 < >>>>>>>>>>>>>>> paul.roewer1990@googlemail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> nice, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> which file system i must use for the cluster? java.io or >>>>>>>>>>>>>>>> hadoop.fs or flink? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2015-06-02 14:29 GMT+02:00 Robert Metzger < >>>>>>>>>>>>>>>> rmetzger@apache.org>: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> you can start Flink on YARN on the Cloudera distribution. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> See here for more: >>>>>>>>>>>>>>>>> http://ci.apache.org/projects/flink/flink-docs-master/set= up/yarn_setup.html >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> These are the commands you need to execute >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wget http://stratosphere-bin.s3-website-us-east-1.amazona= ws.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>>>>>>>>>>>> tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgzcd flink-0.9-S= NAPSHOT/ >>>>>>>>>>>>>>>>> ./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Jun 2, 2015 at 2:03 PM, Pa R=C3=B6 < >>>>>>>>>>>>>>>>> paul.roewer1990@googlemail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> hi community, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> i want test my flink k-means on a hadoop cluster. i use >>>>>>>>>>>>>>>>>> the cloudera live distribution. how i can run flink on t= his cluster? maybe >>>>>>>>>>>>>>>>>> only the java dependencies are engouth? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> best regards, >>>>>>>>>>>>>>>>>> paul >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --047d7b874996e5744d0517b2f754 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
sorry, i see my yarn end before i can run my app, i must s= et the writ= e access for yarn, maybe this solve my problem.

2015-06-04 17:33 GMT+= 02:00 Pa R=C3=B6 <paul.roewer1990@googlemail.com>:
i sta= rt the yarn-session.sh with sudo
and than the flink run command wi= th sudo,
i get the following exception:
=
cloudera@quickstart bin]$ sudo ./flink run /home/cloudera/Desktop/ma-fl= ink.jar
log4j:WARN No appenders could be found for logger (org.apache.h= adoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize = the log4j system properly.
log4j:WARN See http://logging.apache= .org/log4j/1.2/faq.html#noconfig for more info.
org.apache.fl= ink.client.program.ProgramInvocationException: Failed to resolve JobManager=
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program.Client.run(Client= .java:378)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.clien= t.program.Client.run(Client.java:355)
=C2=A0=C2=A0=C2=A0 at org.apache.f= link.client.program.Client.run(Client.java:348)
=C2=A0=C2=A0=C2=A0 at or= g.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment= .java:63)
=C2=A0=C2=A0=C2=A0 at mgm.tp.bigdata.ma_flink.FlinkMain.main(F= linkMain.java:70)
=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMethodAccessor= Impl.invoke0(Native Method)
=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMeth= odAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
=C2=A0=C2=A0=C2= =A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43)
=C2=A0=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(= Method.java:606)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program.P= ackagedProgram.callMainMethod(PackagedProgram.java:437)
=C2=A0=C2=A0=C2= =A0 at org.apache.flink.client.program.PackagedProgram.invokeInteractiveMod= eForExecution(PackagedProgram.java:353)
=C2=A0=C2=A0=C2=A0 at org.apache= .flink.client.program.Client.run(Client.java:315)
=C2=A0=C2=A0=C2=A0 at = org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584)=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.CliFrontend.run(CliFrontend.= java:290)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.CliFrontend.pars= eParameters(CliFrontend.java:880)
=C2=A0=C2=A0=C2=A0 at org.apache.flink= .client.CliFrontend.main(CliFrontend.java:922)
Caused by: java.io= .IOException: JobManager at akka.tcp://flink@127.0.0.1:6123/user/jobmanager<= /a> not reachable. Please make sure that the JobManager is running and its = port is reachable.
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobman= ager.JobManager$.getJobManagerRemoteReference(JobManager.scala:1198)
=C2= =A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobmanager.JobManager$.getJobMa= nagerRemoteReference(JobManager.scala:1222)
=C2=A0=C2=A0=C2=A0 at org.ap= ache.flink.runtime.jobmanager.JobManager$.getJobManagerRemoteReference(JobM= anager.scala:1240)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobman= ager.JobManager.getJobManagerRemoteReference(JobManager.scala)
=C2=A0=C2= =A0=C2=A0 at org.apache.flink.client.program.Client.run(Client.java:375)=C2=A0=C2=A0=C2=A0 ... 15 more
Caused by: akka.actor.ActorNotFound: Act= or not found for: ActorSelection[Anchor(akka.tcp://flink@127.0.0.1:6123/), Path(/user/j= obmanager)]
=C2=A0=C2=A0=C2=A0 at akka.actor.ActorSelection$$anonfun$res= olveOne$1.apply(ActorSelection.scala:65)
=C2=A0=C2=A0=C2=A0 at akka.acto= r.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
= =C2=A0=C2=A0=C2=A0 at scala.concurrent.impl.CallbackRunnable.run(Promise.sc= ala:32)
=C2=A0=C2=A0=C2=A0 at akka.dispatch.BatchingExecutor$Batch$$anon= fun$run$1.processBatch$1(BatchingExecutor.scala:67)
=C2=A0=C2=A0=C2=A0 a= t akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(Batching= Executor.scala:82)
=C2=A0=C2=A0=C2=A0 at akka.dispatch.BatchingExecutor$= Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
=C2=A0=C2=A0=C2=A0= at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecu= tor.scala:59)
=C2=A0=C2=A0=C2=A0 at scala.concurrent.BlockContext$.withB= lockContext(BlockContext.scala:72)
=C2=A0=C2=A0=C2=A0 at akka.dispatch.B= atchingExecutor$Batch.run(BatchingExecutor.scala:58)
=C2=A0=C2=A0=C2=A0 = at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExe= cute(Future.scala:74)
=C2=A0=C2=A0=C2=A0 at akka.dispatch.BatchingExecut= or$class.execute(BatchingExecutor.scala:110)
=C2=A0=C2=A0=C2=A0 at akka.= dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala= :73)
=C2=A0=C2=A0=C2=A0 at scala.concurrent.impl.CallbackRunnable.execut= eWithValue(Promise.scala:40)
=C2=A0=C2=A0=C2=A0 at scala.concurrent.impl= .Promise$DefaultPromise.tryComplete(Promise.scala:248)
=C2=A0=C2=A0=C2= =A0 at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
=C2=A0= =C2=A0=C2=A0 at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:= 508)
=C2=A0=C2=A0=C2=A0 at akka.actor.DeadLetterActorRef.specialHandle(A= ctorRef.scala:541)
=C2=A0=C2=A0=C2=A0 at akka.actor.DeadLetterActorRef.$= bang(ActorRef.scala:531)
=C2=A0=C2=A0=C2=A0 at akka.remote.RemoteActorRe= fProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)=C2=A0=C2=A0=C2=A0 at akka.remote.EndpointWriter.postStop(Endpoint.scala:= 561)
=C2=A0=C2=A0=C2=A0 at akka.actor.Actor$class.aroundPostStop(Actor.s= cala:475)
=C2=A0=C2=A0=C2=A0 at akka.remote.EndpointActor.aroundPostStop= (Endpoint.scala:415)
=C2=A0=C2=A0=C2=A0 at akka.actor.dungeon.FaultHandl= ing$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.s= cala:210)
=C2=A0=C2=A0=C2=A0 at akka.actor.dungeon.FaultHandling$class.t= erminate(FaultHandling.scala:172)
=C2=A0=C2=A0=C2=A0 at akka.actor.Actor= Cell.terminate(ActorCell.scala:369)
=C2=A0=C2=A0=C2=A0 at akka.actor.Act= orCell.invokeAll$1(ActorCell.scala:462)
=C2=A0=C2=A0=C2=A0 at akka.actor= .ActorCell.systemInvoke(ActorCell.scala:478)
=C2=A0=C2=A0=C2=A0 at akka.= dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
=C2=A0=C2= =A0=C2=A0 at akka.dispatch.Mailbox.run(Mailbox.scala:220)<= br>=C2=A0=C2=A0=C2=A0 at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
= =C2=A0=C2=A0=C2=A0 at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoi= nTask.java:260)
=C2=A0=C2=A0=C2=A0 at scala.concurrent.forkjoin.ForkJoin= Pool$WorkQueue.runTask(ForkJoinPool.java:1339)
=C2=A0=C2=A0=C2=A0 at sca= la.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
= =C2=A0=C2=A0=C2=A0 at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(Fo= rkJoinWorkerThread.java:107)

the FlinkMain.java: 70 is= :
env.execute(=
"KMeans Flink");

2015-06-04 17:17 GMT+02:00 Pa R=C3=B6 = <paul.roewer1990@googlemail.com>:
i try this:

[cloudera@qu= ickstart bin]$ sudo su yarn
bash-4.1$ hadoop fs -chmod 777 /user/= cloudera/outputs
chmod: changing permissions of '/user/cloudera/outp= uts': Permission denied. user=3Dyarn is not the owner of inode=3Doutput= s
bash-4.1$ hadoop fs -chmod 777 /user/cloudera/inputs
chmod: changin= g permissions of '/user/cloudera/inputs': Permission denied. user= =3Dyarn is not the owner of inode=3Dinputs
bash-4.1$ exit
exit
[cl= oudera@quickstart bin]$ sudo ./flink run /home/cloudera/Desktop/ma-flink.ja= r
log4j:WARN No appenders could be found for logger (org.apache.ha= doop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize t= he log4j system properly.
log4j:WARN See http://logging.apache.= org/log4j/1.2/faq.html#noconfig for more info.
Found YARN properties= file /home/cloudera/Desktop/flink-0.9-SNAPSHOT/bin/../conf/.yarn-propertie= s
Using JobManager address from YARN properties quickstart.cloudera/127.0.0.1:53874
org.apache.flink.client.program.ProgramInvocationException: The program= execution failed: Failed to submit job 2f46ef5dff4ecf5552b3477ed1c6f4b9 (K= Means Flink)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program= .Client.run(Client.java:412)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.clie= nt.program.Client.run(Client.java:355)
=C2=A0=C2=A0=C2=A0 at org.apache.= flink.client.program.Client.run(Client.java:348)
=C2=A0=C2=A0=C2=A0 at o= rg.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironmen= t.java:63)
=C2=A0=C2=A0=C2=A0 at mgm.tp.bigdata.ma_flink.FlinkMain.main(= FlinkMain.java:70)
=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMethodAccesso= rImpl.invoke0(Native Method)
=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMet= hodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
=C2=A0=C2=A0=C2= =A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce= ssorImpl.java:43)
=C2=A0=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(= Method.java:606)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program.P= ackagedProgram.callMainMethod(PackagedProgram.java:437)
=C2=A0=C2=A0=C2= =A0 at org.apache.flink.client.program.PackagedProgram.invokeInteractiveMod= eForExecution(PackagedProgram.java:353)
=C2=A0=C2=A0=C2=A0 at org.apache= .flink.client.program.Client.run(Client.java:315)
=C2=A0=C2=A0=C2=A0 at = org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584)=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.CliFrontend.run(CliFrontend.= java:290)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.CliFrontend.pars= eParameters(CliFrontend.java:880)
=C2=A0=C2=A0=C2=A0 at org.apache.flink= .client.CliFrontend.main(CliFrontend.java:922)
Caused by: org.apa= che.flink.runtime.client.JobExecutionException: Failed to submit job 2f46ef= 5dff4ecf5552b3477ed1c6f4b9 (KMeans Flink)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobmanager.JobManager.org$apache$f= link$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:595)
=C2= =A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$= receiveWithLogMessages$1.applyOrElse(JobManager.scala:192)
=C2=A0=C2=A0= =C2=A0 at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(Abstr= actPartialFunction.scala:33)
=C2=A0=C2=A0=C2=A0 at scala.runtime.Abstrac= tPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
=C2=A0= =C2=A0=C2=A0 at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(Abstrac= tPartialFunction.scala:25)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.yarn.A= pplicationMasterActor$$anonfun$receiveYarnMessages$1.applyOrElse(Applicatio= nMasterActor.scala:99)
=C2=A0=C2=A0=C2=A0 at scala.PartialFunction$OrEls= e.apply(PartialFunction.scala:162)
=C2=A0=C2=A0=C2=A0 at org.apache.flin= k.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:36)
=C2= =A0=C2=A0=C2=A0 at org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(= ActorLogMessages.scala:29)
=C2=A0=C2=A0=C2=A0 at scala.PartialFunction$c= lass.applyOrElse(PartialFunction.scala:118)
=C2=A0=C2=A0=C2=A0 at org.ap= ache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.sc= ala:29)
=C2=A0=C2=A0=C2=A0 at akka.actor.Actor$class.aroundReceive(Actor= .scala:465)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobmanager.Jo= bManager.aroundReceive(JobManager.scala:94)
=C2=A0=C2=A0=C2=A0 at akka.a= ctor.ActorCell.receiveMessage(ActorCell.scala:516)
=C2=A0=C2=A0=C2=A0 at= akka.actor.ActorCell.invoke(ActorCell.scala:487)
=C2=A0=C2=A0=C2=A0 at = akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
=C2=A0=C2=A0=C2= =A0 at akka.dispatch.Mailbox.run(Mailbox.scala:221)
=C2=A0=C2=A0=C2=A0 a= t akka.dispatch.Mailbox.exec(Mailbox.scala:231)
=C2=A0=C2=A0=C2=A0 at sc= ala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
=C2= =A0=C2=A0=C2=A0 at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask= (ForkJoinPool.java:1339)
=C2=A0=C2=A0=C2=A0 at scala.concurrent.forkjoin= .ForkJoinPool.runWorker(ForkJoinPool.java:1979)
=C2=A0=C2=A0=C2=A0 at sc= ala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:= 107)
Caused by: org.apache.flink.runtime.JobException: Creating t= he input splits caused an error: File /user/cloudera/inputs does not exist = or the user running Flink ('yarn') has insufficient permissions to = access it.
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.executio= ngraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:162)
=C2= =A0=C2=A0=C2=A0 at org.apache.flink.runtime.executiongraph.ExecutionGraph.a= ttachJobGraph(ExecutionGraph.java:471)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runti= me$jobmanager$JobManager$$submitJob(JobManager.scala:535)
=C2=A0=C2=A0= =C2=A0 ... 21 more
Caused by: java.io.FileNotFoundException: File= /user/cloudera/inputs does not exist or the user running Flink ('yarn&= #39;) has insufficient permissions to access it.
=C2=A0=C2=A0=C2= =A0 at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFi= leSystem.java:106)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.api.common.io.= FileInputFormat.createInputSplits(FileInputFormat.java:390)
=C2=A0=C2=A0= =C2=A0 at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(= FileInputFormat.java:51)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.= executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:146)=
=C2=A0=C2=A0=C2=A0 ... 23 more


2015-06-04 17:15 GMT+02:00 = Robert Metzger <rmetzger@apache.org>:
As the output of the "hadoop" tool= indicates, it expects two arguments, you only passed one (777).
The se= cond argument it is expecting is the path to the file you want to change.

In your case, it is:
hadoop fs -chmod 777=C2=A0/user/cloudera/outputs


The reason why=C2=A0
hadoop fs -chmod 777 *
does no= t work is the following: the * is evaluated by your local bash and expanded= to the files which are present in your current, local directory. The bash = expansion is not able to expand to the files in HDFS.

<= /div>

On= Thu, Jun 4, 2015 at 5:08 PM, Pa R=C3=B6 <paul.roewer1990@goo= glemail.com> wrote:
[cloudera@quickstart bin]$ sudo su yarn
bash-4.1$ hadoop = fs -chmod 777
-chmod: Not enough arguments: expected 2 but got 1
Usag= e: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | OCTALMODE&g= t; PATH...
bash-4.1$

you understand?

2015-06-04 17:04 GM= T+02:00 Robert Metzger <rmetzger@apache.org>:
It looks like the user "yarn&qu= ot; which is running Flink doesn't have permission to access the files.=

Can you do "sudo su yarn" to become the "= ;yarn" user. Then, you can do "hadoop fs -chmod 777" to make= the files accessible for everyone.


On Thu, Jun 4, 2015 = at 4:59 PM, Pa R=C3=B6 <paul.roewer1990@googlemail.com>= ; wrote:
oka= y, it's work, i get a exception:

[cloudera@quickstart Desktop]$ = cd flink-0.9-SNAPSHOT/bin/
[cloudera@quickstart bin]$ flink run /home/cl= oudera/Desktop/ma-flink.jar
bash: flink: command not found
[cloudera@= quickstart bin]$ ./flink run /home/cloudera/Desktop/ma-flink.jar
log4j:W= ARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.= MutableMetricsFactory).
log4j:WARN Please initialize the log4j system pr= operly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.= html#noconfig for more info.
Found YARN properties file /home/cloude= ra/Desktop/flink-0.9-SNAPSHOT/bin/../conf/.yarn-properties
Using JobMana= ger address from YARN properties quickstart.cloudera/127.0.0.1:53874
java.io.IOException: = Mkdirs failed to create /user/cloudera/outputs
=C2=A0=C2=A0=C2=A0 at org= .apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438)=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.ChecksumFileSystem.create(Check= sumFileSystem.java:424)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.fs.FileS= ystem.create(FileSystem.java:905)
=C2=A0=C2=A0=C2=A0 at org.apache.hadoo= p.fs.FileSystem.create(FileSystem.java:886)
=C2=A0=C2=A0=C2=A0 at org.ap= ache.hadoop.fs.FileSystem.create(FileSystem.java:783)
=C2=A0=C2=A0=C2=A0= at mgm.tp.bigdata.ma_commons.commons.Seeding.randomSeeding(Seeding.java:21= )
=C2=A0=C2=A0=C2=A0 at mgm.tp.bigdata.ma_flink.FlinkMain.getCentroidDat= aSet(FlinkMain.java:178)
=C2=A0=C2=A0=C2=A0 at mgm.tp.bigdata.ma_flink.F= linkMain.main(FlinkMain.java:47)
=C2=A0=C2=A0=C2=A0 at sun.reflect.Nativ= eMethodAccessorImpl.invoke0(Native Method)
=C2=A0=C2=A0=C2=A0 at sun.ref= lect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
= =C2=A0=C2=A0=C2=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Deleg= atingMethodAccessorImpl.java:43)
=C2=A0=C2=A0=C2=A0 at java.lang.reflect= .Method.invoke(Method.java:606)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.c= lient.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
= =C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program.PackagedProgram.invok= eInteractiveModeForExecution(PackagedProgram.java:353)
=C2=A0=C2=A0=C2= =A0 at org.apache.flink.client.program.Client.run(Client.java:315)
=C2= =A0=C2=A0=C2=A0 at org.apache.flink.client.CliFrontend.executeProgram(CliFr= ontend.java:584)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.CliFronte= nd.run(CliFrontend.java:290)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.clie= nt.CliFrontend.parseParameters(CliFrontend.java:880)
=C2=A0=C2=A0=C2=A0 = at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922)
org.ap= ache.flink.client.program.ProgramInvocationException: The program execution= failed: Failed to submit job 934743a5c49c6d5e31c9e8201452e36d (KMeans Flin= k)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program.Client.run(Clie= nt.java:412)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program.Clien= t.run(Client.java:355)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.pro= gram.Client.run(Client.java:348)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.= client.program.ContextEnvironment.execute(ContextEnvironment.java:63)
= =C2=A0=C2=A0=C2=A0 at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java= :70)
=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(= Native Method)
=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMethodAccessorImp= l.invoke(NativeMethodAccessorImpl.java:57)
=C2=A0=C2=A0=C2=A0 at sun.ref= lect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:= 43)
=C2=A0=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(Method.java:60= 6)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.program.PackagedProgram= .callMainMethod(PackagedProgram.java:437)
=C2=A0=C2=A0=C2=A0 at org.apac= he.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(P= ackagedProgram.java:353)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.p= rogram.Client.run(Client.java:315)
=C2=A0=C2=A0=C2=A0 at org.apache.flin= k.client.CliFrontend.executeProgram(CliFrontend.java:584)
=C2=A0=C2=A0= =C2=A0 at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290)
= =C2=A0=C2=A0=C2=A0 at org.apache.flink.client.CliFrontend.parseParameters(C= liFrontend.java:880)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.client.CliFr= ontend.main(CliFrontend.java:922)
Caused by: org.apache.flink.runtime.cl= ient.JobExecutionException: Failed to submit job 934743a5c49c6d5e31c9e82014= 52e36d (KMeans Flink)
=C2=A0=C2=A0=C2=A0 at org.apache.flin= k.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$Job= Manager$$submitJob(JobManager.scala:595)
=C2=A0=C2=A0=C2=A0 at org.apach= e.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.app= lyOrElse(JobManager.scala:192)
=C2=A0=C2=A0=C2=A0 at scala.runtime.Abstr= actPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)<= br>=C2=A0=C2=A0=C2=A0 at scala.runtime.AbstractPartialFunction$mcVL$sp.appl= y(AbstractPartialFunction.scala:33)
=C2=A0=C2=A0=C2=A0 at scala.runtime.= AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
= =C2=A0=C2=A0=C2=A0 at org.apache.flink.yarn.ApplicationMasterActor$$anonfun= $receiveYarnMessages$1.applyOrElse(ApplicationMasterActor.scala:99)
=C2= =A0=C2=A0=C2=A0 at scala.PartialFunction$OrElse.apply(PartialFunction.scala= :162)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.ActorLogMessages$$a= non$1.apply(ActorLogMessages.scala:36)
=C2=A0=C2=A0=C2=A0 at org.apache.= flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:29)
= =C2=A0=C2=A0=C2=A0 at scala.PartialFunction$class.applyOrElse(PartialFuncti= on.scala:118)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.ActorLogMes= sages$$anon$1.applyOrElse(ActorLogMessages.scala:29)
=C2=A0=C2=A0=C2=A0 = at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
=C2=A0=C2=A0=C2= =A0 at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobMana= ger.scala:94)
=C2=A0=C2=A0=C2=A0 at akka.actor.ActorCell.receiveMessage(= ActorCell.scala:516)
=C2=A0=C2=A0=C2=A0 at akka.actor.ActorCell.invoke(A= ctorCell.scala:487)
=C2=A0=C2=A0=C2=A0 at akka.dispatch.Mailbox.processM= ailbox(Mailbox.scala:254)
=C2=A0=C2=A0=C2=A0 at akka.dispatch.Mailbox.ru= n(Mailbox.scala:221)
=C2=A0=C2=A0=C2=A0 at akka.dispatch.Mailbox.exec(Ma= ilbox.scala:231)
=C2=A0=C2=A0=C2=A0 at scala.concurrent.forkjoin.ForkJoi= nTask.doExec(ForkJoinTask.java:260)
=C2=A0=C2=A0=C2=A0 at scala.concurre= nt.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
=C2= =A0=C2=A0=C2=A0 at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoi= nPool.java:1979)
=C2=A0=C2=A0=C2=A0 at scala.concurrent.forkjoin.ForkJoi= nWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.f= link.runtime.JobException: Creating the input splits caused an error: File = /user/cloudera/outputs/seed-1 does not exist or the user running Flink (= 9;yarn') has insufficient permissions to access it.
=C2=A0=C2=A0=C2= =A0 at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init&= gt;(ExecutionJobVertex.java:162)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.= runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:47= 1)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.runtime.jobmanage= r.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(= JobManager.scala:535)
=C2=A0=C2=A0=C2=A0 ... 21 more
Caused by: java.= io.FileNotFoundException: File /user/cloudera/outputs/seed-1 does not exist= or the user running Flink ('yarn') has insufficient permissions to= access it.
=C2=A0=C2=A0=C2=A0 at org.apache.flink.core.fs.local.LocalFi= leSystem.getFileStatus(LocalFileSystem.java:106)
=C2=A0=C2=A0=C2=A0 at o= rg.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFo= rmat.java:390)
=C2=A0=C2=A0=C2=A0 at org.apache.flink.api.common.io.File= InputFormat.createInputSplits(FileInputFormat.java:51)
=C2=A0=C2=A0=C2= =A0 at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init&= gt;(ExecutionJobVertex.java:146)
=C2=A0=C2=A0=C2=A0 ... 23 more

<= br>
how i must set the files in the hdfs?
quickstart.cloudera:5007= 5/home/cloudera/output?

<= div class=3D"gmail_quote">2015-06-04 16:51 GMT+02:00 Robert Metzger <rme= tzger@apache.org>:
Once you've started the YARN session, you can submit a Flink j= ob with "./bin/flink run <pathToYourJar>".

The jar file of your job doesn't need to be in HDFS. It has to be in= the local file system and flink will send it to all machines.
<= div>

On Thu, = Jun 4, 2015 at 4:48 PM, Pa R=C3=B6 <paul.roewer1990@googlemai= l.com> wrote:
okay, now it run on my hadoop.
how i can start my flink job? an= d where must the jar file save, at hdfs or as local file?

2015-06-0= 4 16:31 GMT+02:00 Robert Metzger <rmetzger@apache.org>:
Yes, you have to run these= commands in the command line of the Cloudera VM.

On Thu, Jun 4, 2015 at 4:28= PM, Pa R=C3=B6 <paul.roewer1990@googlemail.com> wrote:
you mean run th= is command on terminal/shell and not define a hue job?
<= div class=3D"gmail_extra">
2015-06-04 16:25 G= MT+02:00 Robert Metzger <rmetzger@apache.org>:
It should be certainly possible to = run Flink on a cloudera live VM

I think these are the co= mmands you need to execute:

wget http://stratosphere-bin.s3-website-us-east-1.amazonaw= s.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz
tar xvzf flink-0.9-SNAPSHOT= -bin-hadoop2.tgz
cd flink-0.9-SNAPSHOT/
export HADOOP_CONF= _DIR=3D/usr/lib/hadoop/etc/hadoop/
./bin/yarn-session.sh -n 1 -jm 10= 24 -tm 1024

If that is not working for you, please= post the exact error message you are getting and I can help you to get it = to run.


=
On Thu, Jun 4, 2015 at 4:18 PM, Pa R=C3=B6 <paul.roewer1990@googlemail.com> wrote:
hi robert,
=

i think the problem is the hue api,
i had the same = problem with spark submit script,
but on the new hue release, they have= a spark submit api.

i asked the group for the same problem wi= th spark, no reply.

i want test my app on local cluster, before i run it on= the big cluster,
for that i= use cloudera live. maybe it give an other way to test flink on a local clu= ster vm?

2015-06-04 16:12 GMT+02:00 Robert Metzger <rmet= zger@apache.org>:
Hi Paul,

why did running Flink from the regular= scripts not work for you?

I'm not an expert o= n Hue, I would recommend asking in the Hue user forum / mailing list:=C2=A0= https://groups.google.com/a/cloudera.org/forum/#!forum/h= ue-user.

On Thu, Jun 4, 2015 at 4:09 PM, Pa R=C3=B6 <= paul.roewer1990@googlemail.com> wrote:
thanks,
now i want run my= app on cloudera live vm single node,
how i can define my fli= nk job with hue?
i try to run the flink script in the hdfs, it's not= work.

best regards,
paul

2015-06-0= 2 14:50 GMT+02:00 Robert Metzger <rmetzger@apache.org>:
I would recommend using HD= FS.
For that, you need to specify the paths like this: hdfs:///path/to/= data.

On Tue, Jun 2, 2015 at 2:48 PM, Pa R=C3=B6 <= paul.ro= ewer1990@googlemail.com> wrote:
nice,

which file system i must use = for the cluster? java.io o= r hadoop.fs or flink?

2015-06-02 14:29 GMT+02:00 Robert Metzger <rmetz= ger@apache.org>:
Hi,
you can start Flink on YARN on the Cloudera distribution.<= /div>


These are the commands you need to execu= te
wget http://stratosp=
here-bin.s3-website-us-east-1.amazonaws.com/flink-0.9-SNAPSHOT-bin-hadoop2.=
tgz
tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgz
cd flink-0.9-SNAPSHOT/
./bin/yarn-session.sh -n 4 -jm =
1024 -tm 4096

=

<= br>
On Tue, Jun 2, 2015 at 2:03 PM, Pa R=C3=B6 <paul.roewer1990@googlemail.com> wrote:
hi community,
<= br>
i want test my flink k-means on a hadoop cluster. i use the cloude= ra live distribution. how i can run flink on this cluster? maybe only the j= ava dependencies are engouth?

best regards,
paul
<= /div>


















--047d7b874996e5744d0517b2f754--