Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3B9AC18E61 for ; Mon, 26 Oct 2015 14:12:35 +0000 (UTC) Received: (qmail 8323 invoked by uid 500); 26 Oct 2015 14:12:00 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 8245 invoked by uid 500); 26 Oct 2015 14:12:00 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 8236 invoked by uid 99); 26 Oct 2015 14:12:00 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Oct 2015 14:12:00 +0000 Received: from mail-vk0-f52.google.com (mail-vk0-f52.google.com [209.85.213.52]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 701161A0163 for ; Mon, 26 Oct 2015 14:12:00 +0000 (UTC) Received: by vkgy127 with SMTP id y127so99688847vkg.0 for ; Mon, 26 Oct 2015 07:11:59 -0700 (PDT) X-Gm-Message-State: ALoCoQnBgBmycVyk3ViTt2FIEVKlMqU44m55ZQsbAeVfpgjXhDbgX1pdSq/iyBOWweVeSUqXwWbl X-Received: by 10.31.6.19 with SMTP id 19mr24490282vkg.0.1445868719505; Mon, 26 Oct 2015 07:11:59 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.61.133 with HTTP; Mon, 26 Oct 2015 07:11:40 -0700 (PDT) In-Reply-To: References: From: Maximilian Michels Date: Mon, 26 Oct 2015 15:11:40 +0100 Message-ID: Subject: Re: Error running an hadoop job from web interface To: "user@flink.apache.org" Content-Type: multipart/alternative; boundary=001a1143f2566b2f990523028bac --001a1143f2566b2f990523028bac Content-Type: text/plain; charset=UTF-8 That's odd. Does it also execute with parallelism 36 then? On Mon, Oct 26, 2015 at 3:06 PM, Flavio Pompermaier wrote: > No, I just use the default parallelism > > On Mon, Oct 26, 2015 at 3:05 PM, Maximilian Michels > wrote: > >> Did you set the default parallelism of the cluster to 36? This is because >> the plan gets optimized against the cluster configuration when you try to >> run the uploaded program. Before, it doesn't do any optimization. This >> might not be very intuitive. We should probably change that. >> >> On Mon, Oct 26, 2015 at 2:03 PM, Flavio Pompermaier > > wrote: >> >>> Now that I've recompiled flink and restarted the web-client everything >>> works fine. >>> >>> However, when I flag the job I want to run I see parallelism 1 in the >>> right panel, but when I click on "Run Job" button + show optimizer plan >>> flagged I see parallelism 36. Is that a bug of the first preview? >>> >>> >>> On Mon, Oct 26, 2015 at 10:01 AM, Maximilian Michels >>> wrote: >>> >>>> Correct. I'll fix it today. >>>> >>>> Cheers, >>>> Max >>>> >>>> On Mon, Oct 26, 2015 at 9:08 AM, Flavio Pompermaier < >>>> pompermaier@okkam.it> wrote: >>>> >>>>> Running from the shell everything works..is it a problem of >>>>> classloaders hierarchy in the webapp? >>>>> >>>>> On Fri, Oct 23, 2015 at 5:53 PM, Maximilian Michels >>>>> wrote: >>>>> >>>>>> ./bin/flink run /path/to/jar arguments >>>>>> >>>>>> or >>>>>> >>>>>> ./bin/flink run -c MainClass /path/to/jar arguments >>>>>> >>>>>> On Fri, Oct 23, 2015 at 5:50 PM, Stefano Bortoli >>>>> > wrote: >>>>>> >>>>>>> What I normally do is to >>>>>>> >>>>>>> java -cp MYUBERJAR.jar my.package.mainclass >>>>>>> >>>>>>> does it make sense? >>>>>>> >>>>>>> 2015-10-23 17:22 GMT+02:00 Flavio Pompermaier >>>>>>> : >>>>>>> >>>>>>>> could you write ne the command please?I'm not in the office right >>>>>>>> now.. >>>>>>>> On 23 Oct 2015 17:10, "Maximilian Michels" wrote: >>>>>>>> >>>>>>>>> Could you try submitting the job from the command-line and see if >>>>>>>>> it works? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Max >>>>>>>>> >>>>>>>>> On Fri, Oct 23, 2015 at 4:42 PM, Flavio Pompermaier < >>>>>>>>> pompermaier@okkam.it> wrote: >>>>>>>>> >>>>>>>>>> 0.10-snapshot >>>>>>>>>> On 23 Oct 2015 16:09, "Maximilian Michels" >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Flavio, >>>>>>>>>>> >>>>>>>>>>> Which version of Flink are you using? >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Max >>>>>>>>>>> >>>>>>>>>>> On Fri, Oct 23, 2015 at 2:45 PM, Flavio Pompermaier < >>>>>>>>>>> pompermaier@okkam.it> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi to all, >>>>>>>>>>>> I'm trying to run a job from the web interface but I get this >>>>>>>>>>>> error: >>>>>>>>>>>> >>>>>>>>>>>> java.lang.RuntimeException: java.io.FileNotFoundException: JAR entry core-site.xml not found in /tmp/webclient-jobs/EntitonsJsonizer.jar >>>>>>>>>>>> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2334) >>>>>>>>>>>> at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2187) >>>>>>>>>>>> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2104) >>>>>>>>>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:853) >>>>>>>>>>>> at org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2088) >>>>>>>>>>>> at org.apache.hadoop.mapred.JobConf.(JobConf.java:446) >>>>>>>>>>>> at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:175) >>>>>>>>>>>> at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:156) >>>>>>>>>>>> at it.okkam.flink.entitons.io.utils.ParquetThriftEntitons.readEntitons(ParquetThriftEntitons.java:42) >>>>>>>>>>>> at it.okkam.flink.entitons.io.utils.ParquetThriftEntitons.readEntitonsWithId(ParquetThriftEntitons.java:73) >>>>>>>>>>>> at org.okkam.entitons.EntitonsJsonizer.readAtomQuads(EntitonsJsonizer.java:235) >>>>>>>>>>>> at org.okkam.entitons.EntitonsJsonizer.main(EntitonsJsonizer.java:119) >>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497) >>>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395) >>>>>>>>>>>> at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:80) >>>>>>>>>>>> at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:220) >>>>>>>>>>>> at org.apache.flink.client.CliFrontend.info(CliFrontend.java:412) >>>>>>>>>>>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:977) >>>>>>>>>>>> at org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:171) >>>>>>>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) >>>>>>>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) >>>>>>>>>>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) >>>>>>>>>>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) >>>>>>>>>>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) >>>>>>>>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965) >>>>>>>>>>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388) >>>>>>>>>>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187) >>>>>>>>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901) >>>>>>>>>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) >>>>>>>>>>>> at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) >>>>>>>>>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113) >>>>>>>>>>>> at org.eclipse.jetty.server.Server.handle(Server.java:352) >>>>>>>>>>>> at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) >>>>>>>>>>>> at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048) >>>>>>>>>>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) >>>>>>>>>>>> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) >>>>>>>>>>>> at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) >>>>>>>>>>>> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489) >>>>>>>>>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>> Caused by: java.io.FileNotFoundException: JAR entry core-site.xml not found in /tmp/webclient-jobs/EntitonsJsonizer.jar >>>>>>>>>>>> at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:140) >>>>>>>>>>>> at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150) >>>>>>>>>>>> at java.net.URL.openStream(URL.java:1037) >>>>>>>>>>>> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2163) >>>>>>>>>>>> at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2234) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I checked the jar and it contains the core-site.xml file..Am I >>>>>>>>>>>> forced to configure the hadoop classpath in my Flink cluster config files? >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Flavio >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> > --001a1143f2566b2f990523028bac Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
That's odd. Does it also execute with parallelism 36 t= hen?

On = Mon, Oct 26, 2015 at 3:06 PM, Flavio Pompermaier <pompermaier@okkam.it<= /a>> wrote:
No= , I just use the default parallelism

On Mon, Oct 26, 2015 at 3:05 PM, = Maximilian Michels <mxm@apache.org> wrote:
Did you set the default parallelism of = the cluster to 36? This is because the plan gets optimized against the clus= ter configuration when you try to run the uploaded program. Before, it does= n't do any optimization. This might not be very intuitive. We should pr= obably change that.
On Mon, Oct 26, 2015 at 2:03 PM, Flavio Pomperm= aier <pompermaier@okkam.it> wrote:
Now that I've recompiled flink and resta= rted the web-client everything works fine.=C2=A0

However= , when I flag the job I want to run I see parallelism 1 in the right panel,= but when I click on "Run Job" button + show optimizer plan flagg= ed I see parallelism 36. Is that a bug of the first preview?


O= n Mon, Oct 26, 2015 at 10:01 AM, Maximilian Michels <<= a href=3D"mailto:mxm@apache.org" target=3D"_blank">mxm@apache.org> wrote:
C= orrect. I'll fix it today.

Cheers,
Max
=

On Mon,= Oct 26, 2015 at 9:08 AM, Flavio Pompermaier <pompermaier@okkam.it= > wrote:
Runni= ng from the shell everything works..is it a problem of classloaders hierarc= hy in the webapp?

On Fri, Oct 23, 2015 at 5:53 PM, Maximilian Michels <mxm@apache.org= > wrote:
<= div>./bin/flink run /path/to/jar arguments

or

./bin/fl= ink run -c MainClass /path/to/jar arguments

On Fri, Oct 23, 2015 at 5:5= 0 PM, Stefano Bortoli <s.bortoli@gmail.com> wrote:
What I normally do is to <= br>
java -cp MYUBERJAR.jar my.package.mainclass

does it mak= e sense?

2015-10-23 17:22 GMT+02:00 Flavio Pompermaier = <pompermaier@o= kkam.it>:

co= uld you write ne the command please?I'm not in the office right now..

On 23 Oct 2015 17:10, "Maximilian Michels&q= uot; <mxm@apache.org= > wrote:
Could you try submitting the job from the command-li= ne and see if it works?

Thanks,
Max

On Fri, Oct 23, 2015 at 4= :42 PM, Flavio Pompermaier <pompermaier@okkam.it> wrote:<= br>

0.10-snapshot

On 23 Oct 2015 16:09, "Maximilian Michels&q= uot; <mxm@apache.org= > wrote:
Hi Flavio,

Which version of Flink = are you using?

Cheers,
Max

On Fri, Oct 23, 2015 at 2:45 PM, Flavio = Pompermaier <pompermaier@okkam.it> wrote:
Hi to all,
I'm trying to run a = job from the web interface but I get this error:

<= pre style=3D"padding:0px;margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);= text-align:center">java.lang.RuntimeException: java.io.FileNotFoundExceptio= n: JAR entry core-site.xml not found in /tmp/webclient-jobs/EntitonsJsonize= r.jar at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:23= 34) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2= 187) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2104) at org.apache.hadoop.conf.Configuration.get(Configuration.java:853) at org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2= 088) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:446) at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:175) at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:156) at it.okkam.flink.entitons.io.utils.ParquetThriftEntitons.readEntitons(Par= quetThriftEntitons.java:42) at it.okkam.flink.entitons.io.utils.ParquetThriftEntitons.readEntitonsWith= Id(ParquetThriftEntitons.java:73) at org.okkam.entitons.EntitonsJsonizer.readAtomQuads(EntitonsJsonizer.java= :235) at org.okkam.entitons.EntitonsJsonizer.main(EntitonsJsonizer.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja= va:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso= rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.flink.client.program.PackagedProgram.callMainMethod(Packaged= Program.java:497) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeFo= rExecution(PackagedProgram.java:395) at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPl= an(OptimizerPlanEnvironment.java:80) at org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:220= ) at org.apache.flink.client.CliFrontend.info(CliFrontend.java:412) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:97= 7) at org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionSer= vlet.java:171) at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:4= 53) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler= .java:227) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler= .java:965) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:38= 8) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.= java:187) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.= java:901) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.jav= a:117) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47= ) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.j= ava:113) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.ja= va:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(H= ttpConnection.java:1048) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoin= t.java:489) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.j= ava:436) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.FileNotFoundException: JAR entry core-site.xml not found= in /tmp/webclient-jobs/EntitonsJsonizer.jar at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java= :140) at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnecti= on.java:150) at java.net.URL.openStream(URL.java:1037) at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2163) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:22= 34)

I checked the jar and it contains the core-site= .xml file..Am I forced to configure the hadoop classpath in my Flink cluste= r config files?

Best,
Flavio





<= /p>




=



<= /p>


--001a1143f2566b2f990523028bac--