livy-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vanzin <...@git.apache.org>
Subject [GitHub] incubator-livy pull request #118: [LIVY-503] Separate thrift server session ...
Date Wed, 28 Nov 2018 17:20:40 GMT
GitHub user vanzin reopened a pull request:

    https://github.com/apache/incubator-livy/pull/118

    [LIVY-503] Separate thrift server session code in separate module.

    This change creates a new module ("livy-thriftserver-session") with
    the code related to the Thrift server that is expected to run inside
    a Spark session. The module contains a few Job implementations that
    are used by the Thrift server to run SQL statements on the session
    side, plus some internal bookkeeping code.
    
    The new module is a re-implementation of the current Scala code in
    Java; Java is chosen because it becomes simpler to support different
    Scala versions, for a couple of reasons:
    
    - no need to compile the module separately for different Scala versions,
      for when that becomes an issue.
    - better control of serialization, instead of relying on Scala types that
      may not be compatible across versions (imagine the Livy server and the
      Spark session using different Scala versions).
    
    The change also removes the "jarLocation" configuration; in a release
    package, the session jars would be available with the rest of the RSC
    jars; for development, some code was added to the RSC to add the
    thrift server jars to the session when a local build is detected.
    
    Also included is a small change in the RSC pom so that the shaded jar
    is generated in the correct target directory and later copied into the
    staging directory; this solves an issue with using the RSC in test code
    in other modules (such as the one being added).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vanzin/incubator-livy LIVY-503

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-livy/pull/118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #118
    
----
commit 5a033b7c89544d2e3a9b2536b095bae10212e91f
Author: Marcelo Vanzin <vanzin@...>
Date:   2018-09-10T18:32:29Z

    [LIVY-503] Separate thrift server session code in separate module.
    
    This change creates a new module ("livy-thriftserver-session") with
    the code related to the Thrift server that is expected to run inside
    a Spark session. The module contains a few Job implementations that
    are used by the Thrift server to run SQL statements on the session
    side, plus some internal bookkeeping code.
    
    The new module is a re-implementation of the current Scala code in
    Java; Java is chosen because it becomes simpler to support different
    Scala versions, for a couple of reasons:
    
    - no need to compile the module separately for different Scala versions,
      for when that becomes an issue.
    - better control of serialization, instead of relying on Scala types that
      may not be compatible across versions (imagine the Livy server and the
      Spark session using different Scala versions).
    
    The change also removes the "jarLocation" configuration; in a release
    package, the session jars would be available with the rest of the RSC
    jars; for development, some code was added to the RSC to add the
    thrift server jars to the session when a local build is detected.
    
    Also included is a small change in the RSC pom so that the shaded jar
    is generated in the correct target directory and later copied into the
    staging directory; this solves an issue with using the RSC in test code
    in other modules (such as the one being added).

commit f057e4aa322c39c535a1fc9dc86488d09c4b174f
Author: Marcelo Vanzin <vanzin@...>
Date:   2018-10-01T19:58:05Z

    Restore import.

commit 67e0837c6e3f493a84284665a16f8d3d1210d6b2
Author: Marcelo Vanzin <vanzin@...>
Date:   2018-10-25T19:46:28Z

    Merge branch 'master' into LIVY-503

commit 04a3462d0704aca2a89b0493defa8fd26c690a05
Author: Marcelo Vanzin <vanzin@...>
Date:   2018-11-26T16:22:00Z

    Merge branch 'master' into LIVY-503

commit 7ae469485d9682bacc55b936ae3a74c3e644c754
Author: Marcelo Vanzin <vanzin@...>
Date:   2018-11-26T19:04:04Z

    Fix json4s version for Spark 2.4.

commit 212010ea5789d9498f4bd8f4eb6be050f66c2bab
Author: Marcelo Vanzin <vanzin@...>
Date:   2018-11-26T20:53:14Z

    Localize json4s version override.

commit ff5996d4183ff0b41c4097d68c148e8a7c8a70e6
Author: Marcelo Vanzin <vanzin@...>
Date:   2018-11-27T18:21:06Z

    Remove Spark references from DataType.
    
    Otherwise Kryo complains.

----


---

Mime
View raw message