carbondata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yadong Qi (JIRA)" <>
Subject [jira] [Created] (CARBONDATA-998) Don't request executors when we use carbon distribution
Date Thu, 27 Apr 2017 07:47:04 GMT
Yadong Qi created CARBONDATA-998:

             Summary: Don't request executors when we use carbon distribution
                 Key: CARBONDATA-998
             Project: CarbonData
          Issue Type: Improvement
          Components: spark-integration
    Affects Versions: 1.1.0
            Reporter: Yadong Qi
            Assignee: Yadong Qi

In the current implementation, carbon request executors if it need more executors. This brings
the following questions:
1. Carbon interferes with spark dispatch to make request/remove executors confusedly.
2. Carbon will involk DistributionUtil.getDistinctNodesList and sleep some times to make sure
the executors are assigned, it wastes a lot of time.

We'd better make the data scan and schedule independently.

This message was sent by Atlassian JIRA

View raw message