flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@apache.org>
Subject Re: Flink and Clojure
Date Thu, 10 Dec 2015 23:22:00 GMT
Thanks for this idea.

I extended my pom to include clojure-1.5.1.jar in my program jar.
However, the problem is still there... I did some research on the
Internet, and it seems I need to mess around with Clojure's class
loading strategy...

-Matthias

On 12/10/2015 06:47 PM, Nick Dimiduk wrote:
> I think Mattias's project is using maven though -- there's a pom in the
> project that doesn't look generated. If you want to do it from lein, maybe
> my old lein-hadoop [0] plugin can help?
> 
> [0]: https://github.com/ndimiduk/lein-hadoop
> 
> On Thu, Dec 10, 2015 at 8:54 AM, Robert Metzger <rmetzger@apache.org> wrote:
> 
>> I had the same though as Nick. Maybe Leiningen allows to somehow build a
>> fat-jar containing the clojure standard library.
>>
>> On Thu, Dec 10, 2015 at 5:51 PM, Nick Dimiduk <ndimiduk@apache.org> wrote:
>>
>>> What happens when you follow the packaging examples provided in the flink
>>> quick start archetypes? These have the maven-foo required to package an
>>> uberjar suitable for flink submission. Can you try adding that step to
>> your
>>> pom.xml?
>>>
>>> On Thursday, December 10, 2015, Stephan Ewen <sewen@apache.org> wrote:
>>>
>>>> This is a problem in Java.
>>>> I think you cannot dynamically modify the initial system class loader.
>>>>
>>>> What most apps do is check for the thread context class loader when
>>>> dynamically loading classes. We can check and make sure that one is
>> set,
>>>> but if Closure does not respect that, we have a problem.
>>>> Then Closure is not built for dynamic class loading.
>>>>
>>>>
>>>>
>>>> On Thu, Dec 10, 2015 at 5:15 PM, Matthias J. Sax <mjsax@apache.org
>>>> <javascript:;>> wrote:
>>>>
>>>>> Would it make sense (if possible?) for Flink to add the user jar
>>>>> dynamically to it's own classpath so Clojure can find it? Or somehow
>>>>> modify Clojure's class loader?
>>>>>
>>>>> The jars in lib are added to the classpath at startup. This makes it
>>>>> practically impossible to execute a Flink program that is written in
>>>>> Clojure right now...
>>>>>
>>>>>
>>>>> On 12/10/2015 05:09 PM, Aljoscha Krettek wrote:
>>>>>> Clojure is not considering the user-jar when trying to load the
>>> class.
>>>>>>
>>>>>>> On 10 Dec 2015, at 17:05, Matthias J. Sax <mjsax@apache.org
>>>> <javascript:;>> wrote:
>>>>>>>
>>>>>>> Hi Squirrels,
>>>>>>>
>>>>>>> I was playing with a Flink Clojure WordCount example today.
>>>>>>> https://github.com/mjsax/flink-external/tree/master/flink-clojure
>>>>>>>
>>>>>>> After building the project with "mvn package" I tried to submit
it
>>> to
>>>> a
>>>>>>> local cluster. Before I started the cluster, I manually copied
>>>>>>> "clojure-1.5.1.jar" into Flink's lib folder.
>>>>>>>
>>>>>>>> cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar
>>> lib/
>>>>>>>> bin/start-local.sh
>>>>>>>
>>>>>>> However, when submitting the jar, I get an exception:
>>>>>>>
>>>>>>>> bin/flink run -c org.apache.flink.clojure.WordCount
>>>>>>>
>>>>>
>>>>
>>>
>> ~/workspace_flink/flink-external/flink-clojure/target/flink-clojure-0.10.0.jar
>>>>>>>
>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> The program finished with the following exception:
>>>>>>>>
>>>>>>>> org.apache.flink.client.program.ProgramInvocationException:
The
>>>>> program's entry point class 'org.apache.flink.clojure.WordCount'
>> threw
>>> an
>>>>> error during initialization.
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:585)
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:195)
>>>>>>>> at
>>>>>
>> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:784)
>>>>>>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:288)
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>>> at
>> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1103)
>>>>>>>> Caused by: java.lang.ExceptionInInitializerError
>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>> at java.lang.Class.forName(Class.java:278)
>>>>>>>> at
>>>>>
>>>>
>>>
>> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:578)
>>>>>>>> ... 5 more
>>>>>>>> Caused by: java.io.FileNotFoundException: Could not locate
>>>>> org/apache/flink/clojure/WordCount__init.class or
>>>>> org/apache/flink/clojure/WordCount.clj on classpath:
>>>>>>>> at clojure.lang.RT.load(RT.java:443)
>>>>>>>> at clojure.lang.RT.load(RT.java:411)
>>>>>>>> at clojure.core$load$fn__5018.invoke(core.clj:5530)
>>>>>>>> at clojure.core$load.doInvoke(core.clj:5529)
>>>>>>>> at clojure.lang.RestFn.invoke(RestFn.java:408)
>>>>>>>> at clojure.lang.Var.invoke(Var.java:415)
>>>>>>>> at org.apache.flink.clojure.WordCount.<clinit>(Unknown
Source)
>>>>>>>> ... 8 more
>>>>>>>
>>>>>>> I am not sure why the class is not found. It is contained in
the
>> jar
>>>>>>> file. I can fix this error by copying the user jar
>>>>>>> (flink-clojure-0.10.0.jar) into Flink's lib-folder.
>>>>>>>
>>>>>>> It seems, that Flink is not looking into the user-jar when loading
>>>> this
>>>>>>> class. Can anybody explain why?
>>>>>>>
>>>>>>> Thx.
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
> 


Mime
View raw message