carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yadong Qi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CARBONDATA-998) Don't request executors when we use carbon distribution
Date Thu, 27 Apr 2017 07:47:04 GMT
Yadong Qi created CARBONDATA-998:
------------------------------------

             Summary: Don't request executors when we use carbon distribution
                 Key: CARBONDATA-998
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-998
             Project: CarbonData
          Issue Type: Improvement
          Components: spark-integration
    Affects Versions: 1.1.0
            Reporter: Yadong Qi
            Assignee: Yadong Qi


In the current implementation, carbon request executors if it need more executors. This brings
the following questions:
1. Carbon interferes with spark dispatch to make request/remove executors confusedly.
2. Carbon will involk DistributionUtil.getDistinctNodesList and sleep some times to make sure
the executors are assigned, it wastes a lot of time.

We'd better make the data scan and schedule independently.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message