spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kiszk <...@git.apache.org>
Subject [GitHub] spark pull request #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "...
Date Tue, 14 Nov 2017 15:08:53 GMT
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17436#discussion_r150859426
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala
---
    @@ -60,9 +62,15 @@ case class InMemoryTableScanExec(
     
       private lazy val columnarBatchSchema = new StructType(columnIndices.map(i => relationSchema(i)))
     
    +  private lazy val memoryMode = SparkEnv.get.memoryManager.tungstenMemoryMode
    +
       private def createAndDecompressColumn(cachedColumnarBatch: CachedBatch): ColumnarBatch
= {
         val rowCount = cachedColumnarBatch.numRows
    -    val columnVectors = OnHeapColumnVector.allocateColumns(rowCount, columnarBatchSchema)
    +    val columnVectors = if (memoryMode == MemoryMode.ON_HEAP) {
    +      OnHeapColumnVector.allocateColumns(rowCount, columnarBatchSchema)
    +    } else {
    +      OffHeapColumnVector.allocateColumns(rowCount, columnarBatchSchema)
    --- End diff --
    
    Great catch, we need to use `context.addTaskCompletionListener` for close this.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message