Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANC1h_t9LzUekYX85tOkuJ-nEnfvoFDJOi9vS9J3=PDsKbwabQ@mail.gmail.com>
References: <5669A2AD.5010600@apache.org>
 <0EF1168A-78D8-499E-A45A-811A0E599DC8@apache.org>
 <5669A509.3080509@apache.org>
 <CANC1h_tGsvV3VUb-rGiK3T0umTbFnq-Eqg+w0XohkrHosyeahg@mail.gmail.com>
 <CANZa=GvG71JFwuNsbtH9JH4-fR9EGONd1GYwaoPNmY8d6SFQ3g@mail.gmail.com>
 <CAGr9p8BkfttW42i6wN1nKUjx3ax9F38JROB-CjF2v-9vJeXq9A@mail.gmail.com>
 <CANZa=Gt8yY_5W3yc84YYrxKRzvdrYOmgY8U8AFbniaxc-eMyrA@mail.gmail.com>
 <566A0918.6060408@apache.org>
 <CANZa=GvrpGR=eLYve5a6VMDfbMsjVthtTF6g62MO0zkqNne+qw@mail.gmail.com>
 <CANC1h_t9LzUekYX85tOkuJ-nEnfvoFDJOi9vS9J3=PDsKbwabQ@mail.gmail.com>
From: Nick Dimiduk <ndimiduk@apache.org>
Date: Fri, 11 Dec 2015 09:26:56 -0800
Message-ID: 
 <CANZa=GsakDNRU4bME3nHrzAH6YJOmwLTKH4ncN_6fV_eQKkUeA@mail.gmail.com>
Subject: Re: Flink and Clojure
To: dev@flink.apache.org
Content-Type: multipart/alternative; boundary=089e0102def47578e20526a2a239

--089e0102def47578e20526a2a239
Content-Type: text/plain; charset=UTF-8

Ah I see. This explains the issues I had with submitting streaming jobs
that package JDBC drivers. Is there a second in the guide/docs about
classloader considerations with Flink?

On Thu, Dec 10, 2015 at 11:53 PM, Stephan Ewen <sewen@apache.org> wrote:

> Flink's classloading is different from Hadoop's.
>
> In Hadoop, the entire JVM is started with all classes (including the user
> jar) in the classpath already. In Flink, jars are added dymanically, to
> running JVMs with custom class loaders. That way, running worker/master
> processes can accept new jars without restarts. Important for low-latency,
> shells, etc
>
> Flink itself respects these classloaders whenever dynamically looking up a
> class. It may be that Closure is written such that it can only dynamically
> instantiate what is the original classpath.
>
>
>
> On Fri, Dec 11, 2015 at 1:31 AM, Nick Dimiduk <ndimiduk@apache.org> wrote:
>
> > As far as the jvm is concerned, clojure is just another library. You
> should
> > be able to package it up like any other dependency and submit the job.
> > That's always how it worked in Hadoop/MR anyway...
> >
> > On Thu, Dec 10, 2015 at 3:22 PM, Matthias J. Sax <mjsax@apache.org>
> wrote:
> >
> > > Thanks for this idea.
> > >
> > > I extended my pom to include clojure-1.5.1.jar in my program jar.
> > > However, the problem is still there... I did some research on the
> > > Internet, and it seems I need to mess around with Clojure's class
> > > loading strategy...
> > >
> > > -Matthias
> > >
> > > On 12/10/2015 06:47 PM, Nick Dimiduk wrote:
> > > > I think Mattias's project is using maven though -- there's a pom in
> the
> > > > project that doesn't look generated. If you want to do it from lein,
> > > maybe
> > > > my old lein-hadoop [0] plugin can help?
> > > >
> > > > [0]: https://github.com/ndimiduk/lein-hadoop
> > > >
> > > > On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <rmetzger@apache.org
> >
> > > wrote:
> > > >
> > > >> I had the same though as Nick. Maybe Leiningen allows to somehow
> > build a
> > > >> fat-jar containing the clojure standard library.
> > > >>
> > > >> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <ndimiduk@apache.org>
> > > wrote:
> > > >>
> > > >>> What happens when you follow the packaging examples provided in the
> > > flink
> > > >>> quick start archetypes? These have the maven-foo required to
> package
> > an
> > > >>> uberjar suitable for flink submission. Can you try adding that step
> > to
> > > >> your
> > > >>> pom.xml?
> > > >>>
> > > >>> On Thursday, December 10, 2015, Stephan Ewen <sewen@apache.org>
> > wrote:
> > > >>>
> > > >>>> This is a problem in Java.
> > > >>>> I think you cannot dynamically modify the initial system class
> > loader.
> > > >>>>
> > > >>>> What most apps do is check for the thread context class loader
> when
> > > >>>> dynamically loading classes. We can check and make sure that one
> is
> > > >> set,
> > > >>>> but if Closure does not respect that, we have a problem.
> > > >>>> Then Closure is not built for dynamic class loading.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <
> mjsax@apache.org
> > > >>>> <javascript:;>> wrote:
> > > >>>>
> > > >>>>> Would it make sense (if possible?) for Flink to add the user jar
> > > >>>>> dynamically to it's own classpath so Clojure can find it? Or
> > somehow
> > > >>>>> modify Clojure's class loader?
> > > >>>>>
> > > >>>>> The jars in lib are added to the classpath at startup. This makes
> > it
> > > >>>>> practically impossible to execute a Flink program that is written
> > in
> > > >>>>> Clojure right now...
> > > >>>>>
> > > >>>>>
> > > >>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
> > > >>>>>> Clojure is not considering the user-jar when trying to load the
> > > >>> class.
> > > >>>>>>
> > > >>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <mjsax@apache.org
> > > >>>> <javascript:;>> wrote:
> > > >>>>>>>
> > > >>>>>>> Hi Squirrels,
> > > >>>>>>>
> > > >>>>>>> I was playing with a Flink Clojure WordCount example today.
> > > >>>>>>>
> > https://github.com/mjsax/flink-external/tree/master/flink-clojure
> > > >>>>>>>
> > > >>>>>>> After building the project with "mvn package" I tried to submit
> > it
> > > >>> to
> > > >>>> a
> > > >>>>>>> local cluster. Before I started the cluster, I manually copied
> > > >>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
> > > >>>>>>>
> > > >>>>>>>> cp
> ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
> > > >>> lib/
> > > >>>>>>>> bin/start-local.sh
> > > >>>>>>>
> > > >>>>>>> However, when submitting the jar, I get an exception:
> > > >>>>>>>
> > > >>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
> > > >>>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>> ------------------------------------------------------------
> > > >>>>>>>> The program finished with the following exception:
> > > >>>>>>>>
> > > >>>>>>>> org.apache.flink.client.program.ProgramInvocationException:
> The
> > > >>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
> > > >> threw
> > > >>> an
> > > >>>>> error during initialization.
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
> > > >>>>>>>> at
> > > >>>>>
> > > >>
> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
> > > >>>>>>>> at
> org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
> > > >>>>>>>> at
> > > >> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
> > > >>>>>>>> Caused by: java.lang.ExceptionInInitializerError
> > > >>>>>>>> at java.lang.Class.forName0(Native Method)
> > > >>>>>>>> at java.lang.Class.forName(Class.java:278)
> > > >>>>>>>> at
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
> > > >>>>>>>> ... 5 more
> > > >>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate
> > > >>>>> org/apache/flink/clojure/WordCount__init.class or
> > > >>>>> org/apache/flink/clojure/WordCount.clj on classpath:
> > > >>>>>>>> at clojure.lang.RT.load(RT.java:443)
> > > >>>>>>>> at clojure.lang.RT.load(RT.java:411)
> > > >>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
> > > >>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
> > > >>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
> > > >>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
> > > >>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown Source)
> > > >>>>>>>> ... 8 more
> > > >>>>>>>
> > > >>>>>>> I am not sure why the class is not found. It is contained in
> the
> > > >> jar
> > > >>>>>>> file. I can fix this error by copying the user jar
> > > >>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
> > > >>>>>>>
> > > >>>>>>> It seems, that Flink is not looking into the user-jar when
> > loading
> > > >>>> this
> > > >>>>>>> class. Can anybody explain why?
> > > >>>>>>>
> > > >>>>>>> Thx.
> > > >>>>>>>
> > > >>>>>>> -Matthias
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>

--089e0102def47578e20526a2a239--