flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Peel <thomas.p...@euranova.eu>
Subject Re: ClassNotFound exception on a reduce group (Cluster mode)
Date Tue, 16 Jun 2015 08:36:11 GMT
Hi,

Thank you for your help. It was actually an issue between the Flink 
version I used to build my quickstart jar and the Flink version on the 
cluster.

Cheers,
Thomas

Le 16/06/15 10:10, Till Rohrmann a écrit :
> I also tested your code with an additional sink and it is executed on a
> cluster without problems.
>
> Cheers,
> Till
>
> On Tue, Jun 16, 2015 at 10:00 AM Chiwan Park <chiwanpark@icloud.com> wrote:
>
>> Hi. What flink version is running now in cluster?
>> I copied your code and packaging with flink quickstart archetype. There is
>> another error because you don’t add any data sink in the program.
>>
>> Regards,
>> Chiwan Park
>>
>>> On Jun 16, 2015, at 4:17 PM, Thomas Peel <thomas.peel@euranova.eu>
>> wrote:
>>>
>>>
>>> Hi guys,
>>>
>>> I am having troubles with a portion of code that throws an exception
>>> when executed on a cluster (through the web interface) but not when
>>> executed localy.
>>>
>>> The code :
>>>
>>> object DorotheaLassoRegression {
>>>
>>> def main(args: Array[String]): Unit = {
>>>
>>> val env = ExecutionEnvironment.getExecutionEnvironment
>>>
>>> val y = env.readTextFile(
>>> "hdfs://IP_ADDRESS/PATH_TO_FILE/dorothea_train.labels"
>>> ).setParallelism(1).map(x => x.toDouble)
>>>
>>> val Y = y.reduceGroup(iterator => iterator.toArray)
>>> env.execute()
>>> }
>>> }
>>>
>>> The exception:
>>> An error occurred while invoking the program:
>>>
>>> The main method caused an error.
>>>
>>> java.lang.ClassNotFoundException:
>>> eu.enx.flink.paper01.DorotheaLassoRegression$$anonfun$3$$anonfun$4 at
>>> java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
>>> java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
>>> java.security.AccessController.doPrivileged(Native Method) at
>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
>>> java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
>>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
>>> java.lang.Class.forName0(Native Method) at
>>> java.lang.Class.forName(Class.java:270) at
>>>
>> org.apache.flink.api.scala.InnerClosureFinder$$anon$4.visitMethodInsn(ClosureCleaner.scala:317)
>>> at
>>> com.esotericsoftware.reflectasm.shaded.org
>> .objectweb.asm.ClassReader.accept(Unknown
>>> Source) at
>>> com.esotericsoftware.reflectasm.shaded.org
>> .objectweb.asm.ClassReader.accept(Unknown
>>> Source) at
>>>
>> org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:93)
>>> at
>>>
>> org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:113)
>>> at org.apache.flink.api.scala.DataSet.clean(DataSet.scala:123) at
>>> org.apache.flink.api.scala.DataSet$$anon$9.(DataSet.scala:635) at
>>> org.apache.flink.api.scala.DataSet.reduceGroup(DataSet.scala:634) at
>>>
>> eu.enx.flink.paper01.DorotheaLassoRegression$.main(DorotheaLassoRegression.scala:42)
>>> at
>>>
>> eu.enx.flink.paper01.DorotheaLassoRegression.main(DorotheaLassoRegression.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606) at
>>>
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>>> at
>>>
>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>>> at
>>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:229)
>>> at
>>>
>> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:183)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at
>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
>>> at
>>>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
>>> at
>>>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
>>> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
>>> at
>>>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>>> at
>>> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
>>> at
>>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:352) at
>>>
>> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
>>> at
>>>
>> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) at
>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
>>> at
>>> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
>>> at
>>>
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
>>> at
>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Line 42 is the reduceGroup operation. Do you have any ideas ? Is it a
>>> version problem ?
>>>
>>> Thank you for your help, kind regards.
>>> Thomas
>>>
>>> -------------------------
>>> Thomas Peel
>>>
>>> Data Scientist
>>> (M) +32 478 98 93 81
>>>
>>> EURA NOVA
>>>
>>> Rue Emile Francqui, 4
>>>
>>> 1435 Mont-Saint-Guibert
>>>
>>> (T) +32 10 75 02 00
>>>
>>> euranova.eu
>>>
>>>
>>
>>
>>


Mime
View raw message