carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhichao Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores
Date Fri, 27 Oct 2017 02:51:00 GMT

    [ https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221627#comment-16221627
] 

Zhichao  Zhang commented on CARBONDATA-1624:
--------------------------------------------

I can implement this feature.
By the way, I find there are many same code snippets to get 'NUM_CORES_LOADING', but there
is a method called 'getNumberOfCores' in CarbonProperties to get  'NUM_CORES_LOADING' too,
I think we can use method 'CarbonProperties.getNumberOfCores' uniformly to get  'NUM_CORES_LOADING'
.
    
{code:java}
Integer.parseInt(CarbonProperties.getInstance()
          .getProperty(CarbonCommonConstants.NUM_CORES_LOADING,
              CarbonCommonConstants.NUM_CORES_DEFAULT_VAL));
    } catch (NumberFormatException e) {
      thread_pool_size = Integer.parseInt(CarbonCommonConstants.NUM_CORES_DEFAULT_VAL);
    }
{code}

right?


> If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading'
dynamically as per the available executor cores 
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-1624
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1624
>             Project: CarbonData
>          Issue Type: Improvement
>          Components: data-load, spark-integration
>    Affects Versions: 1.3.0
>            Reporter: Zhichao  Zhang
>            Assignee: Zhichao  Zhang
>            Priority: Minor
>
> If we are using carbondata + spark to load data, we can set 
> carbon.number.of.cores.while.loading to the  number of executor cores. 
> For example, when set the number of executor cores to 6, it shows that there are at 
> least 6 cores per node for loading data, so we can set 
> carbon.number.of.cores.while.loading to 6 automatically. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message