carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kumar vishal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CARBONDATA-1318) Concurrent data loading is failing
Date Wed, 19 Jul 2017 18:14:00 GMT

    [ https://issues.apache.org/jira/browse/CARBONDATA-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093548#comment-16093548
] 

kumar vishal commented on CARBONDATA-1318:
------------------------------------------

Reason for this issue is simultaneous running of multiple table data loading. In this case
when Unsafe memory is full then in sort step there is no memory left and hence data loading
is failing. To handle this scenario we can have two manager
1. Working memory(Temp memory for operation like sorting, query processing)
2. sorted data storage memory.
Sorting will be done in working memory. After sorting data will be copied to sorted data storage
memory.
so working memory will be always free. 
Working memory can be also used during query and storing column pages 


> Concurrent data loading is failing
> ----------------------------------
>
>                 Key: CARBONDATA-1318
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1318
>             Project: CarbonData
>          Issue Type: Bug
>            Reporter: kumar vishal
>            Assignee: kumar vishal
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Councurrent data loading is failing as unsafememory manager doesnot have space and it
is throwing outofmemory exception
> When Job is cancelled unsafe memory manager is not clearing the resource 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message