flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aljoscha <...@git.apache.org>
Subject [GitHub] incubator-flink pull request: Bring Scala API up to speed with Jav...
Date Sun, 14 Sep 2014 17:56:41 GMT
GitHub user aljoscha opened a pull request:


    Bring Scala API up to speed with Java API

    First of all, don't try reading central Scala API rewrite commit. It's basically a rewrite
from Scratch, so the diff won't make a lot of sense. Just look at the result.
    The commits before the central commit massage the rest of the system into shape to allow
the rewrite. The commits after change some small things and add examples and ITCases.
    @mbalassi: How's that for working on Sundays... :smile_cat: 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/incubator-flink scala-rework

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #119
commit eae0e1bf3dc1cd34a51efdae835be2bb3c5a53e8
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-08-22T10:24:11Z

    Refactor TupleTypeInfo and add GenericPairComparator
    Now we have TupleTypeInfoBase, TupleSerializerBase, and TupleComparatorBase. They
    are now super classes of TupleTypeInfo and the others.
    Also rename compare on DataInputView to compareSerialized because Scala
    cannot distinguish between the to compare methods for some reason.
    This change is necessary for allowing the Scala API to reuse most of the
    The GenericPairComparator uses the new extractKeys method of
    TypeComparator to compare values of any type. This replaces
    TuplePairComparator and some other special-case pair comparators. This
    is preparatory work for enabling support for Scala Tuples and POJO

commit 320e7f84e004c7aecb7edc39293b910c4205e974
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-08-22T16:17:48Z

    Perform TypeExtraction outside of Java Operators
    Before, FlatMapOperator, GroupReduceOperator, MapOperator, and MapPartitionOperator
    performed the Type extraction themselves while the other Operators had TypeInformation
    parameters. Now the are all unified, which makes it possible to use them from the
    Scala API.
    Also Key extraction for selector functions is moved outside of Keys.java

commit 998050fa17fe7466818ef9b3830b3fbeccdeec1f
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-08-26T14:15:05Z

    Move the Iteration "Operators" to the operators package
    I'm doing this in preparation for the Scala API rework, since then I can just copy over
    the operators package and use them for Scala.

commit 35425515373313640cc85c0b697d3a71754a242d
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-08-26T15:22:18Z

    Make DistinctOperator and Keys use TupleComparatorBase
    Was TupleComparator before which does not work when used from the Scala API.

commit da280502192e555e660ad04797d8533648d2681c
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-02T16:40:17Z

    Change ObjectArrayTypeInfo.getInfoFor to use componentInfo.isTupleType()
    This is necessary because the previous test didn't work for Scala tuples.

commit c20e97534a28db6e5d4ff710c419575d0d1b6b6e
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-02T16:41:53Z

    Refactor Tuple wrapping/unwrapping in Join and CoGroup
    Previously both sides were always wrapped/unwrapped, even if one side
    was a tuple type. This was not compatible with the Scala tuples and is
    also not necessary anymore because of the GenericPairComparator.

commit cbbde178ad98f72283d18e57eb7b151f6dfb821e
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-03T12:50:42Z

    Move RichFunctions to api.common package
    They were in api.java before but they can be used from Scala,
    just like the regular functions.

commit 216479be4b9f4a4ac554df99d16029e12e3caadd
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-04T18:51:43Z

    Rewrite the Scala API as (somewhat) thin Layer on Java API
    Don't bother looking at the diff, this is almost a complete rewrite of
    the previous Scala API. This uses all the work put into the Java API,
    such as TypeInformation, the serializers and comparators and the
    The Scala DataSet and ExecutionEnvironment wrap their respective Java
    equivalents. TypeInformation is generated by a macro that uses
    TypeInformationGen and other macro support classes. The Java
    TypeExtractor is completely bypassed but the TypeInformation and
    sub-classes are created by the Scala type analyzer. There is special
    support for Scala Tuples in the form of ScalaTupleTypeInfo,
    ScalaTupleSerializer, and ScalaTupleComparator.
    This also adds tests to flink-scala that are ports of the tests in
    There are not yet any Scala specific tests in flink-tests. All the
    scala example ITCases are commented out, as well as the examples
    themselves. Those will be uncommented once the examples are ported. This
    will happen in separate commits.

commit f1697dc0f53677c801734b7c23b5647f58f11954
Author: vasia <vasilikikalavri@gmail.com>
Date:   2014-09-09T09:16:59Z

    Connected Components example Scala API rewrite

commit 2693d0418270d253be4439edcbdd9a88ff47ab59
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-09T12:48:27Z

    Add ConnectedComponentsITCase for Scala Example
    Also fix some formatting in the example code

commit 99ef2b8b101212c650f1002dde709811bfb69a78
Author: Kostas Tzoumas <kostas@kostass-mbp.fritz.box>
Date:   2014-09-09T14:45:01Z

    WebLog Analysis example

commit cbca37dc6252aaa0845c7ed663b1d9ead20dfa82
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-09T15:52:12Z

    Add WebLogAnalysisITCase for Scala Example
    Some minor reformatting of example code and scaladoc.

commit 381bb3295bc9cd0b244054143f471361e09d608a
Author: Kostas Tzoumas <kostas@kostass-mbp.fritz.box>
Date:   2014-09-10T14:20:46Z

    Transitive closure Scala example

commit a163c7b814c8cab632def36f74fc1b0d40af456f
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-10T15:59:55Z

    Fix formatting in TransitiveClosureNaive Scala Example
    Also actually use termination criterion in TransitivelClosureNaive
    Java example.

commit 647cc2c9b962264a36e895a89a30938328a0166b
Author: Fabian Hueske <fhueske@apache.org>
Date:   2014-09-05T22:59:44Z

    Added TriangleEnumeration examples for reworked Scala API.

commit 7a236c29d202a5bcc41078313b37d83b24ff1dc7
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-11T09:14:48Z

    Add ITCases for TriangleEnumeration Scala Examples
    Also fix some formatting and make TriangleEnumerationOpt Scala produce the
    same output as the Java version.

commit 9bf4fc0ec37eed51df595a321e62bd092fa236cb
Author: Fabian Hueske <fhueske@apache.org>
Date:   2014-09-11T13:42:29Z

    Adds PageRankBasic Scala example. Removes old Scala examples

commit a97f2076bcafcacd7e8d673e97344dd3bf342f60
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-11T14:14:32Z

    Add logback-test.xml to java8-tests and logback.xml to java-examples

commit 1f24e6bc31b420858c047d2b066239e300c58df2
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-11T15:22:29Z

    Add PageRankITCase for Scala Example
    Also fix formatting in PageRank Scala Example.

commit 97115f173e4703a9ba0746d9149ba9e190ef921f
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-11T15:30:50Z

    Fix formatting in EnumTriangles Scala Examples

commit 34989957131b5ce1c6b857dabbc6640301a792d5
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-11T15:47:08Z

    Fix formatting in some Scala Examples
    Link to ExampleData was not fully qualified.

commit b4b9b742f9832cea30a5dee68557267cd28a6245
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-11T15:54:08Z

    Remove Old/Deprecated Scala Examples and ITCases

commit 167cd646abd0248d8e47ef26f03f1426e97a043f
Author: Kostas Tzoumas <kostas@kostass-mbp.fritz.box>
Date:   2014-09-11T16:25:40Z

    Pi estimation example job in Scala

commit b68438630f583a7f77604d3b44368b0d6fca72ea
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-11T16:47:58Z

    Embellish comments of PiEstimation Scala Example

commit 12ebcf9801b0cda98f4f344c77b0e8c5e7ca903a
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-12T09:38:20Z

    [scala] Add Field Name Key Support for Scala Case Classes
    This does not change the runtime behavious. The key field names are
    mapped to tuple indices at pre-flight time.
    Also extends tests to cover the new feature.

commit d4cfb231554e8f9aedbb42ce8c27421bc83520b3
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-12T13:50:09Z

    Fix typo, getSematicProperties -> getSemanticProperties

commit 183b4b9e1c87fcc0afd111c39ffa5e608ecb208b
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-12T14:25:56Z

    Add SemanticPropertiesTranslationTest for Scala API
    Fix join operator to user proper EquiJoin constructor so that semantic
    properties are actually retrieved.

commit bb0e263977cd43aad5e9c2eb348dc67a837a55a2
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-12T14:26:38Z

    Add Scala API completeness Test
    This is very naive right now, just checks whether a function of the same
    name is available. There are lists and regex patterns of excluded
    functions since not all functions in the Java API need to be available
    in the Scala API.

commit c2b25cc755be9ce6eda3311fda29326b4523fa1b
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2014-09-14T16:25:46Z

    Fix formatting in EnumTrianglesBasic.scala

commit 43425b7599c5bdbf611b92e4de08e5f5feec3cbc
Author: mbalassi <balassi.marton@gmail.com>
Date:   2014-09-13T13:15:17Z

    Added LinearRegression scala example. Removed old BGD example.


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message