flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Eclipse issue after latest dependency rework pr
Date Wed, 01 Oct 2014 16:10:15 GMT
Sorry for the hassle, I know this whole dependency thing is tedious, but I
think it is important that we get our guava out of the way. Otherwise
anyone that uses Flink will be in the trouble that we are with hadoop (that
did not shade its guava).

I had the same issue. In a prior version, we had guava as provided in the
parent pom.

The hadoop dependency (that needs guava) thought guava will be provided and
did not add it, ultimately leading to failures. We need to make sure that a
dependency that depends on guava adds its guava version, while our own code
must not add it (our code must only assume its presence - it is rewritten
to the shaded version anyways).

I am not sure how to simplify that, I tried various things for many many
hours and I did not find a better approach.

The only alternative I could come up with is shading guava in each of our
projects separately, which means that the guava classes will be added to
that project's jar file, in a different name space. Since we have 10+
projects depending on guava, we get guava 10 times into our jars in total.
Not a clean solution as well (although it works, as the class loaders
ignore duplicate classes.

What would be cool is if we had a maven option to "unprovide" guava. The we
could provide in the parent pom, and unprovide for the hadoop dependency.





On Wed, Oct 1, 2014 at 5:43 PM, Márton Balassi <balassi.marton@gmail.com>
wrote:

> I've just updated the streaming fork with Stefan's recent commit
> encapsulating this issue, namely:
>
> https://github.com/mbalassi/incubator-flink/commit/949699dbfe17b62352413769635aed3aaff56100
>
> It solves the problem for the streaming-core project, but running the batch
> WordCount example in Eclipse still produces the following:
>
> Executing WordCount example with built-in default data.
>   Provide parameters to read input data from a file.
>   Usage: WordCount <text path> <result path>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/base/Preconditions
> at
>
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper.<init>(UserCodeObjectWrapper.java:40)
> at
>
> org.apache.flink.api.common.operators.base.GenericDataSourceBase.<init>(GenericDataSourceBase.java:58)
> at
>
> org.apache.flink.api.java.operators.DataSource.translateToDataFlow(DataSource.java:75)
> at
>
> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:82)
> at
>
> org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:117)
> at
>
> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:85)
> at
>
> org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:117)
> at
>
> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:85)
> at
>
> org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:60)
> at
>
> org.apache.flink.api.java.operators.OperatorTranslation.translateToPlan(OperatorTranslation.java:48)
> at
>
> org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:650)
> at
>
> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:48)
> at
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:82)
> Caused by: java.lang.ClassNotFoundException:
> com.google.common.base.Preconditions
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 13 more
>
> Of course one approach is to move
>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <version>${guava.version}</version>
> <scope>provided</scope>
> </dependency>
>
> towards the flink-parent pom, but this ultimately works against shading and
> the hadoop 2.5 version is going to fail as it does here:
>
> https://travis-ci.org/mbalassi/incubator-flink/jobs/36780802
>
> Any suggestions? :)
>
> On Wed, Oct 1, 2014 at 9:40 AM, Gyula Fora <gyfora@apache.org> wrote:
>
> > That worked, thanks.
> >
> > On 30 Sep 2014, at 23:12, Stephan Ewen <sewen@apache.org> wrote:
> >
> > > Yes, I wanted to write a few pointers about such issues.
> > >
> > > We recently shaded the guava dependency, meaning that we have a custom
> > > version of guava where all classes reside in
> > > "org.apache.flink.shaded.com.google" and maven rewrites all our guava
> > > references to that namespace.
> > >
> > > The reason is that guava is a frequently used library and versions are
> > not
> > > all compatible. For example, hadoop 2.5 uses guava 11 and fails if we
> > bring
> > > guava 17 into the classpath. The shading causes our guava version to
> not
> > > conflict, because our shaded version's classes reside different
> > namespace.
> > >
> > > Since Eclipse does not interpret the shading (it only happens in the
> > maven
> > > package phase) you need a regular dependency to guava that is not
> > exported
> > > ("provided" scope)
> > >
> > > Try adding this to the project's pom, it will tell maven to compile and
> > > individually run with the guava lib, but not package it.
> > >
> > > <dependency>
> > > <groupId>com.google.guava</groupId>
> > > <artifactId>guava</artifactId>
> > > <version>${guava.version}</version>
> > > <scope>provided</scope>
> > > </dependency>
> > >
> > >
> > >
> > > On Tue, Sep 30, 2014 at 11:02 PM, Gyula Fóra <gyfora@apache.org>
> wrote:
> > >
> > >> Hey,
> > >>
> > >> We have pulled the dependency rework from the apache master and now we
> > >> cannot get our tests to run in eclipse. With maven from the command
> line
> > >> and also with travis it works perfectly but when I try to run tests
> that
> > >> access the the Configuration class we get the following exception:
> > >>
> > >> java.lang.NoClassDefFoundError: com/google/common/io/BaseEncoding
> > >> at
> > >>
> > >>
> >
> org.apache.flink.configuration.Configuration.setBytes(Configuration.java:358)
> > >> at
> > >> .....
> > >> some stuff here
> > >> .....
> > >>
> > >> Caused by: java.lang.ClassNotFoundException:
> > >> com.google.common.io.BaseEncoding
> > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > >> at java.security.AccessController.doPrivileged(Native Method)
> > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > >> ... 32 more
> > >>
> > >> I literally tried everything to make it work, deleted all maven files,
> > >> reinstalled eclipse etc, but still no luck. Do you experience the same
> > >> issue when running the streaming-core tests in eclipse?
> > >>
> > >> Regards,
> > >> Gyula
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message