Mailing-List: contact issues-help@carbondata.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@carbondata.incubator.apache.org
Date: Mon, 28 Nov 2016 09:23:58 +0000 (UTC)
From: "kumar vishal (JIRA)" <jira@apache.org>
To: issues@carbondata.incubator.apache.org
Message-ID: <JIRA.13023674.1480325013000.376027.1480325038507@Atlassian.JIRA>
In-Reply-To: <JIRA.13023674.1480325013000@Atlassian.JIRA>
References: <JIRA.13023674.1480325013000@Atlassian.JIRA> <JIRA.13023674.1480325013978@arcas>
Subject: [jira] [Created] (CARBONDATA-458)  Improving carbon first time
 query performance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 28 Nov 2016 09:24:02 -0000

kumar vishal created CARBONDATA-458:
---------------------------------------

             Summary:  Improving carbon first time query performance
                 Key: CARBONDATA-458
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-458
             Project: CarbonData
          Issue Type: Improvement
          Components: core, data-load, data-query
            Reporter: kumar vishal
            Assignee: kumar vishal


Improving carbon first time query performance

Reason:
1. As file system cache is cleared file reading will make it slower to read and cache
2. In first time query carbon will have to read the footer from file data file to form the btree
3. Carbon reading more footer data than its required(data chunk)
4. There are lots of random seek is happening in carbon as column data(data page, rle, inverted index) are not stored together.

Solution: 
1. Improve block loading time. This can be done by removing data chunk from blockletInfo and storing only offset and length of data chunk
2. compress presence meta bitset stored for null values for measure column using snappy 
3. Store the metadata and data of a column together and read together this reduces random seek and improve IO


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)