livy-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vanzin <...@git.apache.org>
Subject [GitHub] incubator-livy pull request #121: [LIVY-518][BUILD] Support Spark 2.4
Date Wed, 07 Nov 2018 00:10:18 GMT
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/incubator-livy/pull/121#discussion_r231341354
  
    --- Diff: rsc/src/test/java/org/apache/livy/rsc/TestSparkClient.java ---
    @@ -271,7 +275,7 @@ public void call(LivyClient client) throws Exception {
     
       @Test
       public void testSparkSQLJob() throws Exception {
    -    runTest(true, new TestFunction() {
    +    runTest(true, false, new TestFunction() {
    --- End diff --
    
    I think this deserves some more investigation. It seems to maybe be an issue in Spark,
even if it might not affect Livy outside of tests.
    
    If I re-enable the old test, and comment out these lines, things pass:
    
    ```
          File metaStore = new File("metastore_db");
          if (metaStore.exists()) {
            FileUtils.deleteDirectory(metaStore);
          }
    ```
    
    This suite runs multiple different SparkContext / SparkSession instances in the same JVM.
Each one is supposed to be independent of one another, so the lines above shouldn't cause
problems.
    
    Since they seem to be causing problems, either Livy is not shutting down sessions appropriately,
or Spark is keeping state across different SparkSession initializations. Neither is very good...


---

Mime
View raw message