carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CARBONDATA-229) Array Index of bound exception thrown from dictionary look up while writing sort index file
Date Thu, 08 Sep 2016 08:15:20 GMT

    [ https://issues.apache.org/jira/browse/CARBONDATA-229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15473172#comment-15473172
] 

ASF GitHub Bot commented on CARBONDATA-229:
-------------------------------------------

GitHub user manishgupta88 opened a pull request:

    https://github.com/apache/incubator-carbondata/pull/143

    [CARBONDATA-229] Array Index of bound exception thrown from dictionary look up while writing
sort index file

    Problem: Array Index of bound exception thrown from dictionary look up while writing sort
index file
    
    Analysis: Whenever we load dictionary data into memory, then in case of populating reverse
dictionary object sometimes a chunk which has no value is also getting added to the dictionary
chunk list. This is happening because the logic for dictionary chunk distribution in case
of forward dictionary is not implemented for reverse dictionary and 0 size dictionary chunks
are not getting removed while adding to the list of dictionary chunks.
    
    Solution: Add the same distribution logic we have in forward dictionary for populating
reverse dictionary object
    
    Impact area: Sort index generation

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/manishgupta88/incubator-carbondata dictionary_chunk_addition_issue

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-carbondata/pull/143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #143
    
----
commit 1102eed75692b6c0d4094a6e7f71e83464453d13
Author: manishgupta88 <tomanishgupta18@gmail.com>
Date:   2016-09-08T07:38:56Z

    Problem: Array Index of bound exception thrown from dictionary look up while writing sort
index file
    
    Analysis: Whenever we load dictionary data into memory, then in case of populating reverse
dictionary object sometimes a chunk which has no value is also getting added to the dictionary
chunk list. This is happening because the logic for dictionary chunk distribution in case
of forward dictionary is not implemented for reverse dictionary and 0 size dictionary chunks
are not getting removed while adding to the list of dictionary chunks.
    
    Solution: Add the same distribution logic we have in forward dictionary for populating
reverse dictionary object
    
    Impact area: Sort index generation

----


> Array Index of bound exception thrown from dictionary look up while writing sort index
file
> -------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-229
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-229
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: Manish Gupta
>            Assignee: Manish Gupta
>
> Whenever we load dictionary data into memory, then in case of populating reverse dictionary
object sometimes a chunk which has no value is also getting added to the dictionary chunk
list. This is happening because the logic for dictionary chunk distribution in case of forward
dictionary is not implemented for reverse dictionary and 0 size dictionary chunks are not
getting removed while adding to the list of dictionary chunks.
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> 	at java.util.ArrayList.get(ArrayList.java:429)
> 	at org.apache.carbondata.core.cache.dictionary.DictionaryChunksWrapper.next(DictionaryChunksWrapper.java:92)
> 	at org.apache.carbondata.core.writer.sortindex.CarbonDictionarySortInfoPreparator.prepareDictionarySortModels(CarbonDictionarySortInfoPreparator.java:120)
> 	at org.apache.carbondata.core.writer.sortindex.CarbonDictionarySortInfoPreparator.getDictionarySortInfo(CarbonDictionarySortInfoPreparator.java:51)
> 	at org.apache.carbondata.spark.tasks.SortIndexWriterTask.execute(SortIndexWriterTask.scala:44)
> 	at org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD$$anon$1.<init>(CarbonGlobalDictionaryRDD.scala:387)
> 	at org.apache.carbondata.spark.rdd.CarbonGlobalDictionaryGenerateRDD.compute(CarbonGlobalDictionaryRDD.scala:294)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message