flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@apache.org>
Subject Re: Flink and Clojure
Date Fri, 11 Dec 2015 00:31:59 GMT
As far as the jvm is concerned, clojure is just another library. You should
be able to package it up like any other dependency and submit the job.
That's always how it worked in Hadoop/MR anyway...

On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <mjsax@apache.org> wrote:

> Thanks for this idea.
>
> I extended my pom to include clojure-1.5.1.jar in my program jar.
> However, the problem is still there... I did some research on the
> Internet, and it seems I need to mess around with Clojure's class
> loading strategy...
>
> -Matthias
>
> On 12/10/2015 06:47 PM, Nick Dimiduk wrote:
> > I think Mattias's project is using maven though -- there's a pom in the
> > project that doesn't look generated. If you want to do it from lein,
> maybe
> > my old lein-hadoop [0] plugin can help?
> >
> > [0]: https://github.com/ndimiduk/lein-hadoop
> >
> > On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
> >
> >> I had the same though as Nick. Maybe Leiningen allows to somehow build a
> >> fat-jar containing the clojure standard library.
> >>
> >> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <ndimiduk@apache.org>
> wrote:
> >>
> >>> What happens when you follow the packaging examples provided in the
> flink
> >>> quick start archetypes? These have the maven-foo required to package an
> >>> uberjar suitable for flink submission. Can you try adding that step to
> >> your
> >>> pom.xml?
> >>>
> >>> On Thursday, December 10, 2015, Stephan Ewen <sewen@apache.org> wrote:
> >>>
> >>>> This is a problem in Java.
> >>>> I think you cannot dynamically modify the initial system class loader.
> >>>>
> >>>> What most apps do is check for the thread context class loader when
> >>>> dynamically loading classes. We can check and make sure that one is
> >> set,
> >>>> but if Closure does not respect that, we have a problem.
> >>>> Then Closure is not built for dynamic class loading.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <mjsax@apache.org
> >>>> <javascript:;>> wrote:
> >>>>
> >>>>> Would it make sense (if possible?) for Flink to add the user jar
> >>>>> dynamically to it's own classpath so Clojure can find it? Or somehow
> >>>>> modify Clojure's class loader?
> >>>>>
> >>>>> The jars in lib are added to the classpath at startup. This makes
it
> >>>>> practically impossible to execute a Flink program that is written
in
> >>>>> Clojure right now...
> >>>>>
> >>>>>
> >>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> >>>>>> Clojure is not considering the user-jar when trying to load
the
> >>> class.
> >>>>>>
> >>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <mjsax@apache.org
> >>>> <javascript:;>> wrote:
> >>>>>>>
> >>>>>>> Hi Squirrels,
> >>>>>>>
> >>>>>>> I was playing with a Flink Clojure WordCount example today.
> >>>>>>> https://github.com/mjsax/flink-external/tree/master/flink-clojure
> >>>>>>>
> >>>>>>> After building the project with "mvn package" I tried to
submit it
> >>> to
> >>>> a
> >>>>>>> local cluster. Before I started the cluster, I manually
copied
> >>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
> >>>>>>>
> >>>>>>>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
> >>> lib/
> >>>>>>>> bin/start-local.sh
> >>>>>>>
> >>>>>>> However, when submitting the jar, I get an exception:
> >>>>>>>
> >>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> >>>>>>>
> >>>>>>>
> >>>>>>>> ------------------------------------------------------------
> >>>>>>>> The program finished with the following exception:
> >>>>>>>>
> >>>>>>>> org.apache.flink.client.program.ProgramInvocationException:
The
> >>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
> >> threw
> >>> an
> >>>>> error during initialization.
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> >>>>>>>> at
> >>>>>
> >> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> >>>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> >>>>>>>> at
> >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> >>>>>>>> Caused by: java.lang.ExceptionInInitializerError
> >>>>>>>> at java.lang.Class.forName0(Native Method)
> >>>>>>>> at java.lang.Class.forName(Class.java:278)
> >>>>>>>> at
> >>>>>
> >>>>
> >>>
> >>
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> >>>>>>>> ... 5 more
> >>>>>>>> Caused by: java.io.FileNotFoundException: Could not
locate
> >>>>> org/apache/flink/clojure/WordCount__init.class or
> >>>>> org/apache/flink/clojure/WordCount.clj on classpath:
> >>>>>>>> at clojure.lang.RT.load(RT.java:443)
> >>>>>>>> at clojure.lang.RT.load(RT.java:411)
> >>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> >>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
> >>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> >>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
> >>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown
Source)
> >>>>>>>> ... 8 more
> >>>>>>>
> >>>>>>> I am not sure why the class is not found. It is contained
in the
> >> jar
> >>>>>>> file. I can fix this error by copying the user jar
> >>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> >>>>>>>
> >>>>>>> It seems, that Flink is not looking into the user-jar when
loading
> >>>> this
> >>>>>>> class. Can anybody explain why?
> >>>>>>>
> >>>>>>> Thx.
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message