flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5005) Publish Scala 2.12 artifacts
Date Wed, 12 Apr 2017 20:55:42 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966606#comment-15966606
] 

ASF GitHub Bot commented on FLINK-5005:
---------------------------------------

Github user DieBauer commented on the issue:

    https://github.com/apache/flink/pull/3703
  
    Ok, started looking into the issue a bit more. 
    It seems like it has to do with the new lambda generation in scala 2.12 and not with the
asm library. 
    
    From the scala 2.12-M3 release notes (https://github.com/scala/scala/releases/tag/v2.12.0-M3):

    
    > Java 8 style closure classes
    > 
    > Scala 2.12 emits closures in the same style as Java 8.
    > 
    > For each lambda the compiler generates a method containing the lambda body.
    > At runtime, this method is passed as an argument to the LambdaMetaFactory provided
by the JDK, which creates a closure object.
    > 
    > Compared to Scala 2.11, the new scheme has the advantage that the compiler does not
generate an anonymous class for each lambda anymore.
    > This leads to significantly smaller JAR files.
    
    
    Our ClosureCleaner uses the class name for instantiating the ClassReader, which is used
later on. 
    
    However, since scala2.12 doesn't generate anonymous classes, the file isn't found (null),
therefore we get class not found exception, which make sense now. 
    
    We have to look into how to circumvent/implement this new generation of 'lambdas'. 
    
    A small technical example, the testclass which throwed an exception `AcceptPFTestBase`.

      And then the line containing: `protected val groupedTuples = tuples.groupBy(_._1)`
    Since tuples is a `Dataset` the function that we have to check is `_._1` (an anonymous
function). 
    
    when compiling/executing with scala 2.11 
    we get `class org.apache.flink.api.scala.extensions.base.AcceptPFTestBase$$anonfun$1`
as `cls` in the `ClosureCleaner.getClassReader` method. 
    And this is indeed a file generated by the scala compiler and can be resolved by `ls.getResourceAsStream(className)`.
    
    However when using scala 2.12
    we get `class org.apache.flink.api.scala.extensions.base.AcceptPFTestBase$$Lambda$11/1489743810`
which is not an existing file, and cannot be resolved by `ls.getResourceAsStream(className)`.

    
    
    
    Concluding, according to me, with the new scala 2.12 style with lambdas, the current closurecleaner
doesn't suffice. 
    There is also a Spark issue (https://issues.apache.org/jira/browse/SPARK-14540) regarding
closures in scala 2.12/java8. 
    
    Any thoughts how to proceed?



> Publish Scala 2.12 artifacts
> ----------------------------
>
>                 Key: FLINK-5005
>                 URL: https://issues.apache.org/jira/browse/FLINK-5005
>             Project: Flink
>          Issue Type: Improvement
>          Components: Scala API
>            Reporter: Andrew Roberts
>
> Scala 2.12 was [released|http://www.scala-lang.org/news/2.12.0] today, and offers many
compile-time and runtime speed improvements. It would be great to get artifacts up on maven
central to allow Flink users to migrate to Scala 2.12.0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message