carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravipesala <>
Subject [GitHub] carbondata pull request #1471: [WIP] Datamap FineGrain implementation
Date Mon, 06 Nov 2017 10:04:58 GMT
GitHub user ravipesala opened a pull request:

    [WIP] Datamap FineGrain implementation

    Be sure to do all of the following checklist to help us incorporate 
    your contribution quickly and easily:
     Implemented interfaces for FG datamap and integrated to filterscanner to use the pruned
bitset from FG datamap.
    FG Query flow as follows.
     1.The user can add FG datamap to any table and implement there interfaces.
     2. Any filter query which hits the table with datamap will call prune method of FGdatamap.

     3. The prune method of FGDatamap return list FineGrainBlocklet , these blocklets contain
the information of block, blocklet, page and rowids information as well.
     4. The pruned  blocklets are internally wriitten to file and returns only the block ,
blocklet and filepath  information as part of Splits.
    5. Based on the splits scanrdd schedule the tasks. 
    6. In filterscanner we check the datamapwriterpath from split and reNoteads the bitset
if exists. And pass this bitset as input to it.
    NOTE : This PR depends on
     - [X] Any interfaces changed?
           Few changes to datamap interfaces only.
     - [X] Any backward compatibility impacted? NO
     - [X] Document update required? 
         Yes, Required to add to dev guide about interfaces
     - [X] Testing done
            Tests are added for both CG and FG
     - [X] For large changes, please consider breaking it into sub-tasks under an umbrella

You can merge this pull request into a Git repository by running:

    $ git pull datamap-fg-impl

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1471
commit 8d727ffbc465191040274318a55dd0a69063bdfd
Author: ravipesala <>
Date:   2017-10-11T13:37:22Z

    Added FG interfaces

commit 0cc36e4b28d2747728013116cdacecb80beb0239
Author: ravipesala <>
Date:   2017-10-17T12:23:09Z

    Refactored code to remove path from blocklet

commit 3c204c2d2e69512f21f630a091df756255ccd6e8
Author: ravipesala <>
Date:   2017-10-31T09:55:57Z

    fgdatamap implement

commit 717ba3e41a6f3d1936ff14857a9b992cff13c8a6
Author: ravipesala <>
Date:   2017-11-03T16:52:33Z

    Added test for FGDatamap

commit e6093540c800c2e86c472787603c884a72e1d129
Author: ravipesala <>
Date:   2017-11-04T06:21:21Z

    Added testcase



View raw message