spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Herman Schistad (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-9775) Query Mesos for number of CPUs to set default parallelism
Date Sun, 09 Aug 2015 17:37:46 GMT
Herman Schistad created SPARK-9775:
--------------------------------------

             Summary: Query Mesos for number of CPUs to set default parallelism
                 Key: SPARK-9775
                 URL: https://issues.apache.org/jira/browse/SPARK-9775
             Project: Spark
          Issue Type: Improvement
          Components: Mesos
    Affects Versions: 1.4.1
            Reporter: Herman Schistad
            Priority: Minor


As highlighted in a TODO on line 400 of MesosSchedulerBackend.scala (at least on 3ca995b7)
we should query the Mesos master and set the default parallelism based on the number of CPUs
available in the cluster (and multiply by two or three?)

See code in question [here (gitweb)|https://git-wip-us.apache.org/repos/asf?p=spark.git;a=blob;f=core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala;h=3f63ec1c;hb=HEAD#l400].

This task should also update the documentation [here (gitweb)|https://git-wip-us.apache.org/repos/asf?p=spark.git;a=blob;f=docs/configuration.md;h=c60dd1;hb=HEAD#l789]
to highlight the fact.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message