Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@carbondata.apache.org
From: ravipesala <git@git.apache.org>
To: issues@carbondata.apache.org
Reply-To: issues@carbondata.apache.org
Message-ID: <git-pr-1436-carbondata@git.apache.org>
Subject: [GitHub] carbondata pull request #1436: [WIP][CARBONDATA-1617] Merging carbonindex fi...
Content-Type: text/plain
Date: Wed, 25 Oct 2017 14:59:12 +0000 (UTC)
archived-at: Wed, 25 Oct 2017 14:59:15 -0000

GitHub user ravipesala opened a pull request:

    https://github.com/apache/carbondata/pull/1436

    [WIP][CARBONDATA-1617] Merging carbonindex files within segment

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [X ] Make sure the PR title is formatted like:
       `[CARBONDATA-<Jira issue #>] Description of pull request`
       
     - [ X] Make sure to add PR description including
            
           Problem :
    The first-time query of carbon becomes very slow. It is because of reading many small carbonindex files and cache to the driver at the first time. 
    Many carbonindex files are created in below case
    Loading data in large cluster
    For example, if the cluster size is 100 nodes then for each load 100 index files are created per segment. So after 100 loads, the number of carbonindex files becomes 10000. .
    It will be slower to read all the files from the driver since a lot of namenode calls and IO operations.
    Solution :
    Merge the carbonindex files in two levels.so that we can reduce the IO calls to namenode and improves the read performance.
    Merge within a segment.
    Merge the carbonindex files to single file immediately after load completes within the segment. It would be named as a .carbonindexmerge file. It is actually not a true data merging but a simple file merge. So that the current structure of carbonindex files does not change. While reading we just read one file instead of many carbonindex files within the segment.
    
    
     - [ ] Any interfaces changed?
     
     - [ ] Any backward compatibility impacted?
     
     - [ ] Document update required?
    
     - [ ] Testing done
     
            Please provide details on 
            - Whether new unit test cases have been added or why no new tests are required?
            - How it is tested? Please attach test report.
            - Is it a performance related change? Please attach the performance test report.
            - Any additional information to help reviewers in testing this change.
             
     - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. 
                     
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ravipesala/incubator-carbondata carbon-index-merge

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/1436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1436
    
----
commit bbabb705b8dc81a6569b66ee3bb1765de04348c7
Author: ravipesala <ravi.pesala@gmail.com>
Date:   2017-10-25T05:43:22Z

    Merging carbonindex files within segment

----


---