spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dongjoon-hyun <...@git.apache.org>
Subject [GitHub] spark pull request #20590: [SPARK-23399][SQL] Register a task completion lis...
Date Tue, 13 Feb 2018 09:28:24 GMT
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20590#discussion_r167803208
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
---
    @@ -160,6 +162,25 @@ abstract class OrcSuite extends OrcTest with BeforeAndAfterAll {
           }
         }
       }
    +
    +  // This should be tested manually because it raises OOM intentionally
    +  // in order to cause `Leaked filesystem connection`. The test suite dies, too.
    +  ignore("SPARK-23399 Register a task completion listner first for OrcColumnarBatchReader")
{
    +    withSQLConf(SQLConf.ORC_VECTORIZED_READER_BATCH_SIZE.key -> s"${Int.MaxValue}")
{
    +      withTempDir { dir =>
    +        val basePath = dir.getCanonicalPath
    +        Seq(0).toDF("a").write.format("orc").save(new Path(basePath, "first").toString)
    +        Seq(1).toDF("a").write.format("orc").save(new Path(basePath, "second").toString)
    +        val df = spark.read.orc(
    +          new Path(basePath, "first").toString,
    +          new Path(basePath, "second").toString)
    +        val e = intercept[SparkException] {
    +          df.collect()
    +        }
    +        assert(e.getCause.isInstanceOf[OutOfMemoryError])
    +      }
    +    }
    +  }
    --- End diff --
    
    Hi, All. 
    The above test case generates the same leakage reported in JIRA.
    And, this PR fixes that. Please try this in IntelliJ with the original code.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message