carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gvram...@apache.org
Subject [1/2] incubator-carbondata git commit: Problem: Array Index of bound exception thrown from dictionary look up while writing sort index file
Date Thu, 08 Sep 2016 09:21:24 GMT
Repository: incubator-carbondata
Updated Branches:
  refs/heads/master 0b02d9506 -> b7a8ab0f2


Problem: Array Index of bound exception thrown from dictionary look up while writing sort
index file

Analysis: Whenever we load dictionary data into memory, then in case of populating reverse
dictionary object sometimes a chunk which has no value is also getting added to the dictionary
chunk list. This is happening because the logic for dictionary chunk distribution in case
of forward dictionary is not implemented for reverse dictionary and 0 size dictionary chunks
are not getting removed while adding to the list of dictionary chunks.

Solution: Add the same distribution logic we have in forward dictionary for populating reverse
dictionary object

Impact area: Sort index generation


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/9030923e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/9030923e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/9030923e

Branch: refs/heads/master
Commit: 9030923e377d4b2a919799ce689aebfff1adbb76
Parents: 0b02d95
Author: manishgupta88 <tomanishgupta18@gmail.com>
Authored: Thu Sep 8 13:08:56 2016 +0530
Committer: Venkata Ramana G <ramana.gollamudi@huawei.com>
Committed: Thu Sep 8 14:46:20 2016 +0530

----------------------------------------------------------------------
 .../core/cache/dictionary/DictionaryCacheLoaderImpl.java       | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/9030923e/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java
b/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java
index e798c01..ce3107b 100644
--- a/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java
+++ b/core/src/main/java/org/apache/carbondata/core/cache/dictionary/DictionaryCacheLoaderImpl.java
@@ -95,7 +95,7 @@ public class DictionaryCacheLoaderImpl implements DictionaryCacheLoader
{
     int dictionaryChunkSize = CarbonUtil.getDictionaryChunkSize();
     int sizeOfLastDictionaryChunk = dictionaryInfo.getSizeOfLastDictionaryChunk();
     int sizeOfOneDictionaryChunk = dictionaryChunkSize - sizeOfLastDictionaryChunk;
-    if (sizeOfOneDictionaryChunk == 0) {
+    if (sizeOfOneDictionaryChunk <= 0) {
       sizeOfOneDictionaryChunk = dictionaryChunkSize;
     }
     List<List<byte[]>> dictionaryChunks =
@@ -111,7 +111,9 @@ public class DictionaryCacheLoaderImpl implements DictionaryCacheLoader
{
       }
     }
     for (List<byte[]> dictionaryChunk : dictionaryChunks) {
-      dictionaryInfo.addDictionaryChunk(dictionaryChunk);
+      if (!dictionaryChunk.isEmpty()) {
+        dictionaryInfo.addDictionaryChunk(dictionaryChunk);
+      }
     }
   }
 


Mime
View raw message