carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravindra Pesala (JIRA)" <>
Subject [jira] [Created] (CARBONDATA-1522) 6. Loading aggregation tables for streaming data tables.
Date Wed, 27 Sep 2017 10:22:00 GMT
Ravindra Pesala created CARBONDATA-1522:

             Summary: 6. Loading aggregation tables for streaming data tables.
                 Key: CARBONDATA-1522
             Project: CarbonData
          Issue Type: Sub-task
            Reporter: Ravindra Pesala

 we can finish the segment load after receives configurable amount of data and create a new
segment to load new streaming data. While finishing the segment we can trigger the agg tables
on it. So there would not be any agg tables on ongoing streaming segment but querying can
be done on the streaming segment of actual table.
For example user configures stream_segment size as 1 GB, so for every 1 GB of stream data
it receives it creates a new segment and finishes the current segment. While finishing the
current segment we can trigger agg table loading and compaction of segments.
While querying of data we change the query plan to apply union of agg table and streaming
segment of actual table to get the current data.

This message was sent by Atlassian JIRA

View raw message