carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From manishgupta88 <...@git.apache.org>
Subject [GitHub] carbondata pull request #1077: [CARBONDATA-1213] Removed rowCountPercentage ...
Date Thu, 22 Jun 2017 09:24:39 GMT
GitHub user manishgupta88 opened a pull request:

    https://github.com/apache/carbondata/pull/1077

    [CARBONDATA-1213] Removed rowCountPercentage check and fixed IUD data load issue

    Problems:
    1. Row count percentage not required with high cardinality threshold check
    2. IUD returning incorrect results in case of update on high cardinality column
    
    Analysis:
    1. In case a column is identified as high cardinality column still it is not getting converted
to no dictionary column because of another parameter check called rowCountPercentage. Default
value of rowCountPercentage is 80%. Due to this even though high cardinality column is identified,
if it is less than 80% of the total number of rows it will be treated as dictionary column.
This can still lead to executor lost failure due to memory constraints.
    2. RLE on a column is not being set correctly and due to incorrect code design RLE applicable
on a column is decided by a different part of code from the one which is actually applying
the RLE on a column. Because of this Footer is getting filled with incorrect RLE information
and query is failing.
    
    Fix:
    1. Remove an unwanted check for rowCountPercentage.
    2. RLE applicability on a column should be decided from a common place in the code.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/manishgupta88/incubator-carbondata high_cardinlaity_identification_fix

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1077.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1077
    
----
commit 9c1291dc84b8fc4247a9d6e32d4482685d40325a
Author: manishgupta88 <tomanishgupta18@gmail.com>
Date:   2017-06-22T09:07:13Z

    Problem:
    1. Row count percentage not required with high cardinality threshold check
    2. IUD returning incorrect results in case of update on high cardinality column
    
    Analysis:
    1. In case a column is identified as high cardinality column still it is not getting converted
to no dictionary column because of another parameter check called rowCountPercentage. Default
value of rowCountPercentage is 80%. Due to this even though high cardinality column is identified,
if it is less than 80% of the total number of rows it will be treated as dictionary column.
This can still lead to executor lost failure due to memory constraints.
    2. RLE on a column is not being set correctly and due to incorrect code design RLE applicable
on a column is decided by a different part of code from the one which is actually applying
the RLE on a column. Because of this Footer is getting filled with incorrect RLE information
and query is failing.
    
    Fix:
    1. Remove an unwanted check for rowCountPercentage.
    2. RLE applicability on a column should be decided from a common place in the code.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message