flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prez Cannady <revp...@correlatesystems.com>
Subject Re: Results of testing Flink quickstart against 0.10-SNAPSHOT and 1.0-SNAPSHOT (re. Dependency on non-existent org.scalamacros:quasiquotes_2.11:<any version>)
Date Wed, 20 Jan 2016 15:00:08 GMT
Morning, Robert.

You’re right; the 1.0-SNAPSHOT with fetched binaries issue is resolved now.  Unfortunately,
it now emits the same error as 0.10-SNAPSHOT with fetched binaries.  There is a fix for that:

https://github.com/apache/flink/pull/1511 <https://github.com/apache/flink/pull/1511>

It’s definitely in the release-0.10 and master branches

https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java
<https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java>
https://github.com/apache/flink/blob/release-0.10/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java
<https://github.com/apache/flink/blob/release-0.10/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java>

I grabbed the sources jar for the latest 1.0-SNAPSHOT and 0.10-SNAPSHOT:

From https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/0.10-SNAPSHOT/flink-runtime_2.11-0.10-20160114.200924-161-sources.jar
<https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/0.10-SNAPSHOT/flink-runtime_2.11-0.10-20160114.200924-161-sources.jar>

    /**
     * Gets the Flink (internal) accumulators values.
     * @return the serialized map
     */
    public Map<AccumulatorRegistry.Metric, Accumulator<?, ?>> deserializeFlinkAccumulators()
throws IOException, ClassNotFoundException {
        return flinkAccumulators.deserializeValue(ClassLoader.getSystemClassLoader());
    }

ClassLoader.getSystemClassLoader() should be getClass().getClassLoader().

Not sure why it’s not taking in the build, but there’s the problem.

Prez Cannady  
p: 617 500 3378  
e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  









> On Jan 20, 2016, at 8:17 AM, Robert Metzger <rmetzger@apache.org> wrote:
> 
> Hi Prez,
> 
> thanks a lot for the thorough research you did on this issue. The issue with "1.0-SNAPSHOT
with fetched binary dependencies" should be resolved by a fix I've pushed to master yesterday:
> 
> a) The "change-scala-version" script wasn't adopted to the renamed examples directory,
that's why it failed renaming the artifacts for _2.11. That's why the maven dependencies ended
up being mixed between Scala 2.11 and 2.10.
> https://github.com/apache/flink/commit/8f0c47df092ccdc6028dbd88aed197edcd8945ee#diff-c1ca4095c51fcc58f380c59cfabffc8a
<https://github.com/apache/flink/commit/8f0c47df092ccdc6028dbd88aed197edcd8945ee#diff-c1ca4095c51fcc58f380c59cfabffc8a>
> 
> b) The deployment of the scala 2.11 build failed because the "flink-table" module contained
java code in the scala/ directory. Also the code was not compliant with our java checkstyle.

> https://github.com/apache/flink/commit/018f1fee9c856c66454155043006d6f53d39517a <https://github.com/apache/flink/commit/018f1fee9c856c66454155043006d6f53d39517a>
> 
> I hope with these two fixes, the scala 2.11 problems are resolved for now.
> 
> Regards,
> Robert
> 
> On Tue, Jan 19, 2016 at 6:43 AM, Prez Cannady <revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>>
wrote:
> One correction, line 3 of 1.0-SHAPSHOT source build should read “checked out master
branch (snapshot version 1.0-SNAPSHOT)."
> 
> Prez Cannady  
> p: 617 500 3378  
> e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
> LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez> 

> 
> 
> 
> 
> 
> 
> 
> 
> 
>> On Jan 19, 2016, at 12:41 AM, Prez Cannady <revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>>
wrote:
>> 
>> Sent this to dev@flink.apache.org <mailto:dev@flink.apache.org>, but that might
not be the appropriate forum for it.
>> 
>> Finally got a chance to sit down and run through a few tests. Upfront, I have been
able to resolve my issue sufficiently to move forward, but seems there’s an issue with the
current bits for both 1.0-SNAPSHOT and 0.10-SNAPSHOT in the remote Maven repos.
>> 
>> Notes
>> 
>> wordcount-processing <https://github.com/OCExercise/wordcount-processing> is
a customized version of the Flink quickstart archetype I’m using to test Flink integration
with Spring Boot. It is instrumented for Maven and Gradle build and execution.
>> I’m targeting Scala 2.11 and Flink 0.10.
>> 0.10-SNAPSHOT source build
>> 
>> Steps
>> 
>> Checked out release–0.10 branch (snapshot version 0.10-SNAPSHOT).
>> Built with mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dscala.version=2.11.7
-Dscala.binary.version=2.11.
>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>> Result
>> 
>> Maven execution of test succeeds without incident.
>> Gradle execution of test succeeds without incident.
>> 0.10-SNAPSHOT with fetched binary dependencies
>> 
>> Steps
>> 
>> Cleaned out local maven repository with rm -rf $HOME/.m2/repo/org/apache/flink &&
rm -rf $HOME/.m2/repo/org/scala*.
>> Cleaned out local gradle repository with rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.apache.flink
&& rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.scala*.
>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>> Result
>> 
>> Maven build completed without incident. Maven execution error’d out with issue
supposedly resolved with pull request 1511 <https://github.com/apache/flink/pull/1511>.
>> Gradle execution of test succeeds without incident.
>> 1.0-SNAPSHOT source build
>> 
>> Steps
>> 
>> Cleaned out local maven repository with rm -rf $HOME/.m2/repo/org/apache/flink &&
rm -rf $HOME/.m2/repo/org/scala*.
>> Cleaned out local gradle repository with rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.apache.flink
&& rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.scala*.
>> Checked out release–0.10 branch (snapshot version 0.10-SNAPSHOT).
>> Built with mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dscala.version=2.11.7
-Dscala.binary.version=2.11.
>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>> Result
>> 
>> Maven build completed without incident. Maven execution error’d out with issue
supposedly resolved with pull request 1511 <https://github.com/apache/flink/pull/1511>.
>> Gradle execution of test succeeds without incident.
>> 1.0-SNAPSHOT with fetched binary dependencies
>> 
>> Steps
>> 
>> Cleaned out local maven repository with rm -rf $HOME/.m2/repo/org/apache/flink &&
rm -rf $HOME/.m2/repo/org/scala*.
>> Cleaned out local gradle repository with rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.apache.flink
&& rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.scala*.
>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>> Result
>> 
>> Maven build error’d out with
>> could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[String],
and
>> can't expand macros compiled by previous versions of Scala[ERROR] val text = env.fromElements("To
be, or not to be,--that is the question:—“.
>> (Gist <https://gist.github.com/revprez/c70a6f3976b091413c8c>).
>> Gradle build error’d out with java.lang.NoClassDefFoundError: scala/reflect/macros/Context
(gist <https://gist.github.com/revprez/298a0302711908975add>).
>> Discussion
>> 
>> As it stands, I can move forward with building 0.10-SNAPSHOT from source for my own
purposes.
>> I’m guessing pull request 1511 <https://github.com/apache/flink/pull/1511>
hasn’t made it into the the upstream snapshot repos yet.
>> I’m not sure as to why 1.0-SNAPSHOT is suffering from a macros incident and not
0.10-SNAPSHOT. Both are bringing in dependencies (quasiquotes, scala-library, scala-reflect,
and a bunch of akka-* stuff) compiled for 2.10, but 0.10-SNAPSHOT does not invite the macro-related
errors some of us have seen when building against 1.0-SNAPSHOT dependencies.
>> 
>> Prez Cannady  
>> p: 617 500 3378  
>> e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
>> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
>> LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 


Mime
View raw message