Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 688AF200D50 for ; Fri, 27 Oct 2017 08:01:25 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 672AF1609E5; Fri, 27 Oct 2017 06:01:25 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id ABFB7160BF3 for ; Fri, 27 Oct 2017 08:01:24 +0200 (CEST) Received: (qmail 22229 invoked by uid 500); 27 Oct 2017 06:01:23 -0000 Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.apache.org Delivered-To: mailing list issues@carbondata.apache.org Received: (qmail 22208 invoked by uid 99); 27 Oct 2017 06:01:23 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Oct 2017 06:01:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id DFEC31A182C for ; Fri, 27 Oct 2017 06:01:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id j4OeSN4SDBy9 for ; Fri, 27 Oct 2017 06:01:22 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 408215FAF3 for ; Fri, 27 Oct 2017 06:01:22 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 4DA70E2648 for ; Fri, 27 Oct 2017 06:01:20 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 887102130D for ; Fri, 27 Oct 2017 06:01:17 +0000 (UTC) Date: Fri, 27 Oct 2017 06:01:17 +0000 (UTC) From: "Ravindra Pesala (JIRA)" To: issues@carbondata.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 27 Oct 2017 06:01:25 -0000 [ https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221758#comment-16221758 ] Ravindra Pesala commented on CARBONDATA-1624: --------------------------------------------- Welcome to contribute. We should not use the carbonproperties anymore for this dynamic cores as it impacts other loads. First find the available cores which we can allocate for loading per executor before submitting and pass the same information to carbon in RDD compute. > If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores > ---------------------------------------------------------------------------------------------------------------------------------------- > > Key: CARBONDATA-1624 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1624 > Project: CarbonData > Issue Type: Improvement > Components: data-load, spark-integration > Affects Versions: 1.3.0 > Reporter: Zhichao Zhang > Assignee: Zhichao Zhang > Priority: Minor > > If we are using carbondata + spark to load data, we can set > carbon.number.of.cores.while.loading to the number of executor cores. > For example, when set the number of executor cores to 6, it shows that there are at > least 6 cores per node for loading data, so we can set > carbon.number.of.cores.while.loading to 6 automatically. -- This message was sent by Atlassian JIRA (v6.4.14#64029)