carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravipesala <...@git.apache.org>
Subject [GitHub] carbondata pull request #1089: [CARBONDATA-1224] Added page level reader ins...
Date Sat, 24 Jun 2017 02:58:17 GMT
GitHub user ravipesala opened a pull request:

    https://github.com/apache/carbondata/pull/1089

    [CARBONDATA-1224] Added page level reader instead of reading whole blocklet in V3.

    Problem 
    In V3 format we read the whole blocklet at once to memory in order save IO time. But it
turns out to be costlier in case of parallel reading of more carbondata files. 
    For example if we need to compact 50 segments then compactor need to open the readers
on all the 50 segments to do merge sort. But the memory consumption is too high if each reader
reads whole blocklet to the memory and there is high chances of going out of memory.
    
    Solution:
    In this type of scenarios we can introduce new readers for V3 to read the data page by
page instead of reading whole blocklet at once to reduce the memory footprint.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata pagelevel-reader

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1089.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1089
    
----
commit 82e40556cbc9ff7fa872cfc0f39ba6998965acc2
Author: ravipesala <ravi.pesala@gmail.com>
Date:   2017-06-24T02:46:07Z

    Added page by page reader instead of reading whole blocklet.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message