carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yadong Qi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CARBONDATA-1109) Page lost in load process when last page is not be consumed at the end
Date Wed, 31 May 2017 08:12:04 GMT
Yadong Qi created CARBONDATA-1109:
-------------------------------------

             Summary: Page lost in load process when last page is not be consumed at the end
                 Key: CARBONDATA-1109
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1109
             Project: CarbonData
          Issue Type: Bug
            Reporter: Yadong Qi


First, we use Producer-Consumer model in the write step, always we have n(default value is
2 and can be configured) producers and one consumer. The task of generate last page(less than
32000) is added to thread pool at the end, but can't be guaranteed to be finished and add
to BlockletDataHolder at the end. Because we have n tasks running concurrently.
Second, we have 2 ways to invoke `writeDataToFile`, one is the size of `DataWriterHolder`
reach the size of blocklet and two is page is the last page.
So if the last page is not be consumed at the end, we lost the page which be consumed after
last page.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message