flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasiliki Kalavri <vasilikikala...@gmail.com>
Subject Re: Inconvenient (unforeseen?) consequences of PR #1683
Date Mon, 29 Feb 2016 21:57:00 GMT
/me likes this idea :)

On 29 February 2016 at 12:38, Stephan Ewen <sewen@apache.org> wrote:

> How about adding to "flink-gelly-examples" that it packages the examples in
> JAR files. We can then add them even to the "examples/gelly" folder in the
> binary distribution.
>
> On Mon, Feb 29, 2016 at 12:24 PM, Vasiliki Kalavri <
> vasilikikalavri@gmail.com> wrote:
>
> > In my opinion, the fat jar solution is easier than having to copy the
> Gelly
> > jar to all task managers.
> > I would be in favor of including flink-gelly in the flink-examples
> module,
> > but we can also simply document the current behavior nicely.
> >
> > -V.
> >
> > On 29 February 2016 at 12:15, Till Rohrmann <trohrmann@apache.org>
> wrote:
> >
> > > Good catch :-) I mean we could also change the behaviour to include
> > > flink-gelly in the flink-gelly-examples module.
> > >
> > > On Mon, Feb 29, 2016 at 12:13 PM, Vasiliki Kalavri <
> > > vasilikikalavri@gmail.com> wrote:
> > >
> > > > Thanks Till! Then, we'd better update the docs, too. Currently both
> > links
> > > > to examples and cluster execution instructions are broken.
> > > > I'll create an issue.
> > > >
> > > > -V.
> > > >
> > > > On 29 February 2016 at 11:54, Till Rohrmann <trohrmann@apache.org>
> > > wrote:
> > > >
> > > > > Hi Vasia,
> > > > >
> > > > > that is because you're missing the flink-gelly dependency. If you
> > just
> > > > > build flink-gelly-examples, then it won't contain flink-gelly
> because
> > > it
> > > > is
> > > > > not a fat jar. You either have to install flink-gelly on your
> cluster
> > > or
> > > > > package it in the final user jar.
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> > > > > vasilikikalavri@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi squirrels,
> > > > > >
> > > > > > sorry I've been slow to respond to this, but I'm now testing
RC1
> > and
> > > > I'm
> > > > > a
> > > > > > bit confused with this change.
> > > > > >
> > > > > > So far, the easier way to run a Gelly example on a cluster was
to
> > > > package
> > > > > > and submit the Gelly jar.
> > > > > > Now, since the flink-gelly project doesn't contain the examples
> > > > anymore,
> > > > > I
> > > > > > tried running a Gelly example by the packaging
> flink-gelly-examples
> > > jar
> > > > > > (mvn package). However, this gives me a ClassNotFoundException.
> > > > > > e.g. for SingleSourceShortestPaths, the following:
> > > > > >
> > > > > > java.lang.RuntimeException: Could not look up the main(String[])
> > > method
> > > > > > from the class
> > > > org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > > > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
> > > > > > at
> > > >
> org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > > > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > > > > > at
> org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > > > > > Caused by: java.lang.NoClassDefFoundError:
> > > > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > > > > > at java.lang.Class.getMethod0(Class.java:2774)
> > > > > > at java.lang.Class.getMethod(Class.java:1663)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > > > > > ... 5 more
> > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > org.apache.flink.graph.spargel.MessagingFunction
> > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > > > > > ... 10 more
> > > > > >
> > > > > >
> > > > > > What am I missing here?
> > > > > >
> > > > > > Thanks!
> > > > > > -Vasia.
> > > > > >
> > > > > > On 26 February 2016 at 14:10, Márton Balassi <
> > > balassi.marton@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks, +1.
> > > > > > >
> > > > > > > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <
> sewen@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > I think it is a release blocker.
> > > > > > > >
> > > > > > > > That was a change to make it possible to use Flink
with SBT
> > (and
> > > > make
> > > > > > it
> > > > > > > > easier to use it with Maven).
> > > > > > > > Just had an offline chat with Till, and we suggest
the
> > following
> > > > > > > solution:
> > > > > > > >
> > > > > > > > All libraries should be split into two projects:
> > > > > > > >   - library
> > > > > > > >   - library-examples
> > > > > > > >
> > > > > > > > The "library" project has the core flink dependencies
as
> > > "provided"
> > > > > > > > The "library-examples" project has both the "library"
and the
> > > core
> > > > > > flink
> > > > > > > > dependencies with scope "compile"
> > > > > > > >
> > > > > > > > That way the example should run in the IDE out of
the cox,
> and
> > > > users
> > > > > > that
> > > > > > > > reference the libraries will still get the correct
packaging
> > > > (include
> > > > > > the
> > > > > > > > library in the user jar, but not additionally the
core flink
> > > jars).
> > > > > > > >
> > > > > > > > Greetings,
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > > > > > > balassi.marton@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Issued JIRA ticket 3511 to make it referable
in other
> > > > discussions.
> > > > > > [1]
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > > > > > > > >
> > > > > > > > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi
<
> > > > > > > > balassi.marton@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Recent changes to the build [1] where many
libraries got
> > > their
> > > > > core
> > > > > > > > > > dependencies (the ones included in the flink-dist
fat
> jar)
> > > > moved
> > > > > to
> > > > > > > the
> > > > > > > > > > provided scope.
> > > > > > > > > >
> > > > > > > > > > The reasoning was that when submitting to
the Flink
> cluster
> > > the
> > > > > > > > > > application already has these dependencies,
while when a
> > user
> > > > > > writes
> > > > > > > a
> > > > > > > > > > program against these libraries she will
include the core
> > > > > > > dependencies
> > > > > > > > > > explicitly anyway.
> > > > > > > > > >
> > > > > > > > > > There is one other case of usage however,
namely when
> > someone
> > > > is
> > > > > > > trying
> > > > > > > > > to
> > > > > > > > > > run an application defined in these libraries
depending
> on
> > > the
> > > > > core
> > > > > > > > jars.
> > > > > > > > > > To give an example if you were to run the
Gelly
> > > > > ConnectedComponents
> > > > > > > > > example
> > > > > > > > > > [2] from an IDE after importing Flink (or
running with
> java
> > > > -jar
> > > > > > > > without
> > > > > > > > > > including the flink fat jar in the classpath)
you would
> > > receive
> > > > > the
> > > > > > > > > > following class not found exception as per
the current
> > > master:
> > > > > > > > > >
> > > > > > > > > > Exception in thread "main"
> java.lang.NoClassDefFoundError:
> > > > > > > > > > org/apache/flink/api/common/ProgramDescription
> > > > > > > > > > at java.lang.ClassLoader.defineClass1(Native
Method)
> > > > > > > > > > at
> java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > > > > > > > at
> > > java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > > > > > > > at
> > java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > > > > > > > at java.security.AccessController.doPrivileged(Native
> > Method)
> > > > > > > > > > at
> > java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> > > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > > > > > at
> > > > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > > > > > at
> > > > > > >
> > > com.intellij.rt.execution.application.AppMain.main(AppMain.java:122)
> > > > > > > > > >
> > > > > > > > > > Where the class missing in runtime is in
flink-core, a
> > > > transitive
> > > > > > > > > > dependency of the jars moved to the provided
scope.
> > > > > > > > > >
> > > > > > > > > > Funny thing is we have tests in place to
run our
> examples,
> > > but
> > > > > > those
> > > > > > > > add
> > > > > > > > > > test scope dependencies re-adding the missing
classes, so
> > it
> > > is
> > > > > > never
> > > > > > > > > > discovered.
> > > > > > > > > >
> > > > > > > > > > I agree with the original purpose of PR
#1683, but also
> > think
> > > > > that
> > > > > > > the
> > > > > > > > > > current state makes for very inconvenient
user
> experience.
> > > > > > > > > >
> > > > > > > > > > I would like to open a discussion on how
and when to
> > resolve
> > > > the
> > > > > > > issue
> > > > > > > > > > given the release of 1.0.0.
> > > > > > > > > >
> > > > > > > > > > 1. Is it a release blocker?
> > > > > > > > > > 2. Should the change be reverted or is it
sufficient to
> > have
> > > > > proper
> > > > > > > > > > documentation around it? Maybe a maven profile
for
> > explicitly
> > > > for
> > > > > > > > > > developing Flink without the provided scope?
> > > > > > > > > >
> > > > > > > > > > Note that the issue was originally reported
by Gábor
> Gévay.
> > > > > > > > > >
> > > > > > > > > > [1] https://github.com/apache/flink/pull/1683
> > > > > > > > > > [2]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Marton
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message