flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prez Cannady <revp...@correlatesystems.com>
Subject Re: Results of testing Flink quickstart against 0.10-SNAPSHOT and 1.0-SNAPSHOT (re. Dependency on non-existent org.scalamacros:quasiquotes_2.11:<any version>)
Date Thu, 21 Jan 2016 16:54:04 GMT
Apologies for the late reply; been on the road.

I’ve been blowing away my Maven and Gradle repos for each test to make sure I’m pulling
down the latest snapshots.  I’ve also looked into the source jars directly from the snapshot
repo:

1.0-SNAPSHOT: https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/1.0-SNAPSHOT/
<https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/1.0-SNAPSHOT/>
0.10-SNAPSHOT: https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/0.10-SNAPSHOT/
<https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/0.10-SNAPSHOT/>

I note that the jars date to January 14, which may be right on the cusp of when this commit
was taken (7 days ago).  And while the change you and I first discussed made it (https://github.com/apache/flink/commit/0ae46b596949808f56c40bd7a68f478bc10206ab
<https://github.com/apache/flink/commit/0ae46b596949808f56c40bd7a68f478bc10206ab>),
the second did not (https://github.com/apache/flink/commit/cfcb5d7ba5c22295c0ae628f8d9a2447e2286985
<https://github.com/apache/flink/commit/cfcb5d7ba5c22295c0ae628f8d9a2447e2286985>).

Perhaps there’s a fresher repo I should be using?

Prez Cannady  
p: 617 500 3378  
e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  









> On Jan 20, 2016, at 3:48 PM, Stephan Ewen <sewen@apache.org> wrote:
> 
> Hi Prez!
> 
> I merged the pull request into master a while back. Have a look here (https://github.com/apache/flink/commits/master
<https://github.com/apache/flink/commits/master> commits of January 15th).
> 
> Is it possible that you are using a cached older version?
> 
> Greetings,
> Stephan
> 
> 
> 
> 
> On Wed, Jan 20, 2016 at 4:00 PM, Prez Cannady <revprez@correlatesystems.com <mailto:revprez@correlatesystems.com>>
wrote:
> Morning, Robert.
> 
> You’re right; the 1.0-SNAPSHOT with fetched binaries issue is resolved now.  Unfortunately,
it now emits the same error as 0.10-SNAPSHOT with fetched binaries.  There is a fix for that:
> 
> https://github.com/apache/flink/pull/1511 <https://github.com/apache/flink/pull/1511>
> 
> It’s definitely in the release-0.10 and master branches
> 
> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java
<https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java>
> https://github.com/apache/flink/blob/release-0.10/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java
<https://github.com/apache/flink/blob/release-0.10/flink-runtime/src/main/java/org/apache/flink/runtime/accumulators/AccumulatorSnapshot.java>
> 
> I grabbed the sources jar for the latest 1.0-SNAPSHOT and 0.10-SNAPSHOT:
> 
> From https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/0.10-SNAPSHOT/flink-runtime_2.11-0.10-20160114.200924-161-sources.jar
<https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/0.10-SNAPSHOT/flink-runtime_2.11-0.10-20160114.200924-161-sources.jar>
> 
>     /**
>      * Gets the Flink (internal) accumulators values.
>      * @return the serialized map
>      */
>     public Map<AccumulatorRegistry.Metric, Accumulator<?, ?>> deserializeFlinkAccumulators()
throws IOException, ClassNotFoundException {
>         return flinkAccumulators.deserializeValue(ClassLoader.getSystemClassLoader());
>     }
> 
> ClassLoader.getSystemClassLoader() should be getClass().getClassLoader().
> 
> Not sure why it’s not taking in the build, but there’s the problem.
> 
> Prez Cannady  
> p: 617 500 3378  
> e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
> LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez> 

> 
> 
> 
> 
> 
> 
> 
> 
> 
>> On Jan 20, 2016, at 8:17 AM, Robert Metzger <rmetzger@apache.org <mailto:rmetzger@apache.org>>
wrote:
>> 
>> Hi Prez,
>> 
>> thanks a lot for the thorough research you did on this issue. The issue with "1.0-SNAPSHOT
with fetched binary dependencies" should be resolved by a fix I've pushed to master yesterday:
>> 
>> a) The "change-scala-version" script wasn't adopted to the renamed examples directory,
that's why it failed renaming the artifacts for _2.11. That's why the maven dependencies ended
up being mixed between Scala 2.11 and 2.10.
>> https://github.com/apache/flink/commit/8f0c47df092ccdc6028dbd88aed197edcd8945ee#diff-c1ca4095c51fcc58f380c59cfabffc8a
<https://github.com/apache/flink/commit/8f0c47df092ccdc6028dbd88aed197edcd8945ee#diff-c1ca4095c51fcc58f380c59cfabffc8a>
>> 
>> b) The deployment of the scala 2.11 build failed because the "flink-table" module
contained java code in the scala/ directory. Also the code was not compliant with our java
checkstyle. 
>> https://github.com/apache/flink/commit/018f1fee9c856c66454155043006d6f53d39517a <https://github.com/apache/flink/commit/018f1fee9c856c66454155043006d6f53d39517a>
>> 
>> I hope with these two fixes, the scala 2.11 problems are resolved for now.
>> 
>> Regards,
>> Robert
>> 
>> On Tue, Jan 19, 2016 at 6:43 AM, Prez Cannady <revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>>
wrote:
>> One correction, line 3 of 1.0-SHAPSHOT source build should read “checked out master
branch (snapshot version 1.0-SNAPSHOT)."
>> 
>> Prez Cannady  
>> p: 617 500 3378  
>> e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
>> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
>> LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Jan 19, 2016, at 12:41 AM, Prez Cannady <revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>>
wrote:
>>> 
>>> Sent this to dev@flink.apache.org <mailto:dev@flink.apache.org>, but that
might not be the appropriate forum for it.
>>> 
>>> Finally got a chance to sit down and run through a few tests. Upfront, I have
been able to resolve my issue sufficiently to move forward, but seems there’s an issue with
the current bits for both 1.0-SNAPSHOT and 0.10-SNAPSHOT in the remote Maven repos.
>>> 
>>> Notes
>>> 
>>> wordcount-processing <https://github.com/OCExercise/wordcount-processing>
is a customized version of the Flink quickstart archetype I’m using to test Flink integration
with Spring Boot. It is instrumented for Maven and Gradle build and execution.
>>> I’m targeting Scala 2.11 and Flink 0.10.
>>> 0.10-SNAPSHOT source build
>>> 
>>> Steps
>>> 
>>> Checked out release–0.10 branch (snapshot version 0.10-SNAPSHOT).
>>> Built with mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dscala.version=2.11.7
-Dscala.binary.version=2.11.
>>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>>> Result
>>> 
>>> Maven execution of test succeeds without incident.
>>> Gradle execution of test succeeds without incident.
>>> 0.10-SNAPSHOT with fetched binary dependencies
>>> 
>>> Steps
>>> 
>>> Cleaned out local maven repository with rm -rf $HOME/.m2/repo/org/apache/flink
&& rm -rf $HOME/.m2/repo/org/scala*.
>>> Cleaned out local gradle repository with rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.apache.flink
&& rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.scala*.
>>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>>> Result
>>> 
>>> Maven build completed without incident. Maven execution error’d out with issue
supposedly resolved with pull request 1511 <https://github.com/apache/flink/pull/1511>.
>>> Gradle execution of test succeeds without incident.
>>> 1.0-SNAPSHOT source build
>>> 
>>> Steps
>>> 
>>> Cleaned out local maven repository with rm -rf $HOME/.m2/repo/org/apache/flink
&& rm -rf $HOME/.m2/repo/org/scala*.
>>> Cleaned out local gradle repository with rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.apache.flink
&& rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.scala*.
>>> Checked out release–0.10 branch (snapshot version 0.10-SNAPSHOT).
>>> Built with mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dscala.version=2.11.7
-Dscala.binary.version=2.11.
>>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>>> Result
>>> 
>>> Maven build completed without incident. Maven execution error’d out with issue
supposedly resolved with pull request 1511 <https://github.com/apache/flink/pull/1511>.
>>> Gradle execution of test succeeds without incident.
>>> 1.0-SNAPSHOT with fetched binary dependencies
>>> 
>>> Steps
>>> 
>>> Cleaned out local maven repository with rm -rf $HOME/.m2/repo/org/apache/flink
&& rm -rf $HOME/.m2/repo/org/scala*.
>>> Cleaned out local gradle repository with rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.apache.flink
&& rm -rf $HOME/.gradle/caches/modules-2/files-2.1/org.scala*.
>>> Ran wordcount-process with mvn clean spring-boot:run -Drun.arguments=“localhost,9999”.
>>> Ran wordcount-process with gradle bootRun -Drun.arguments=“localhost 9999”.
>>> Result
>>> 
>>> Maven build error’d out with
>>> could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[String],
and
>>> can't expand macros compiled by previous versions of Scala[ERROR] val text =
env.fromElements("To be, or not to be,--that is the question:—“.
>>> (Gist <https://gist.github.com/revprez/c70a6f3976b091413c8c>).
>>> Gradle build error’d out with java.lang.NoClassDefFoundError: scala/reflect/macros/Context
(gist <https://gist.github.com/revprez/298a0302711908975add>).
>>> Discussion
>>> 
>>> As it stands, I can move forward with building 0.10-SNAPSHOT from source for
my own purposes.
>>> I’m guessing pull request 1511 <https://github.com/apache/flink/pull/1511>
hasn’t made it into the the upstream snapshot repos yet.
>>> I’m not sure as to why 1.0-SNAPSHOT is suffering from a macros incident and
not 0.10-SNAPSHOT. Both are bringing in dependencies (quasiquotes, scala-library, scala-reflect,
and a bunch of akka-* stuff) compiled for 2.10, but 0.10-SNAPSHOT does not invite the macro-related
errors some of us have seen when building against 1.0-SNAPSHOT dependencies.
>>> 
>>> Prez Cannady  
>>> p: 617 500 3378  
>>> e: revprez@opencorrelate.org <mailto:revprez@opencorrelate.org>  
>>> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
 
>>> LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>
 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 


Mime
View raw message