carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravindra Pesala (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores
Date Fri, 27 Oct 2017 11:05:00 GMT

    [ https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222176#comment-16222176
] 

Ravindra Pesala commented on CARBONDATA-1624:
---------------------------------------------

Ok, I got your point. You mean just get the configured spark.executor.cores from spark while
initializing session and set to carbon properties.
I was thinking little more about it, get the currently available cores from spark for every
load and block those cores to control the parallel loads.  Any way we can do your way first
later we can extend to get cores for each load.

> If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading'
dynamically as per the available executor cores 
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-1624
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1624
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: data-load, spark-integration
>    Affects Versions: 1.3.0
>            Reporter: Zhichao  Zhang
>            Assignee: Zhichao  Zhang
>            Priority: Minor
>
> If we are using carbondata + spark to load data, we can set 
> carbon.number.of.cores.while.loading to the  number of executor cores. 
> For example, when set the number of executor cores to 6, it shows that there are at 
> least 6 cores per node for loading data, so we can set 
> carbon.number.of.cores.while.loading to 6 automatically. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message