carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gvram...@apache.org
Subject [1/2] incubator-carbondata git commit: Problem: Dictionary cache not getting cleared after task completion in dictionary decoder
Date Fri, 04 Nov 2016 12:20:46 GMT
Repository: incubator-carbondata
Updated Branches:
  refs/heads/master f53d323fa -> ac62b58d7


Problem: Dictionary cache not getting cleared after task completion in dictionary decoder

Analysis: Currently LRU cache eviction policy is based on dictionary access count. For cache
to remove a entry its access count must be 0. In dictionary decoder after conversion of surrogate
key to actual value the access count for dictionary columns in query is not getting decremented
due to which it will never be cleared from memory when LRU cache size is configured.

Fix: Add a task completion listener which will take care of clearing the dictionary in case
of both success and failure

Impact area: LRU cache eviction policy which can lead to query and data load failure


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/7402645d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/7402645d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/7402645d

Branch: refs/heads/master
Commit: 7402645dacea9fd215f7f27b2c21615cf218a867
Parents: f53d323
Author: manishgupta88 <tomanishgupta18@gmail.com>
Authored: Thu Nov 3 21:18:03 2016 +0530
Committer: Venkata Ramana G <ramana.gollamudi@huawei.com>
Committed: Fri Nov 4 15:10:32 2016 +0530

----------------------------------------------------------------------
 .../org/apache/spark/sql/CarbonDictionaryDecoder.scala   | 11 +++++++++++
 1 file changed, 11 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/7402645d/integration/spark/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala
----------------------------------------------------------------------
diff --git a/integration/spark/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala
b/integration/spark/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala
index 12cdf9d..e0a8757 100644
--- a/integration/spark/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala
+++ b/integration/spark/src/main/scala/org/apache/spark/sql/CarbonDictionaryDecoder.scala
@@ -17,6 +17,7 @@
 
 package org.apache.spark.sql
 
+import org.apache.spark.TaskContext
 import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.errors.attachTree
@@ -169,6 +170,16 @@ case class CarbonDictionaryDecoder(
           val dicts: Seq[Dictionary] = getDictionary(absoluteTableIdentifiers,
             forwardDictionaryCache)
           val dictIndex = dicts.zipWithIndex.filter(x => x._1 != null).map(x => x._2)
+          // add a task completion listener to clear dictionary that is a decisive factor
for
+          // LRU eviction policy
+          val dictionaryTaskCleaner = TaskContext.get
+          dictionaryTaskCleaner.addTaskCompletionListener(context =>
+            dicts.foreach { dictionary =>
+              if (null != dictionary) {
+                dictionary.clear
+              }
+            }
+          )
           new Iterator[InternalRow] {
             val unsafeProjection = UnsafeProjection.create(output.map(_.dataType).toArray)
             var flag = true


Mime
View raw message