spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-4962) Put TaskScheduler.start back in SparkContext to shorten cluster resources occupation period
Date Mon, 18 May 2015 18:21:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-4962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-4962:
-----------------------------------

    Assignee: Apache Spark

> Put TaskScheduler.start back in SparkContext to shorten cluster resources occupation
period
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-4962
>                 URL: https://issues.apache.org/jira/browse/SPARK-4962
>             Project: Spark
>          Issue Type: Improvement
>          Components: Scheduler
>    Affects Versions: 1.0.0
>            Reporter: YanTang Zhai
>            Assignee: Apache Spark
>            Priority: Minor
>
> When SparkContext object is instantiated, TaskScheduler is started and some resources
are allocated from cluster. However, these
> resources may be not used for the moment. For example, DAGScheduler.JobSubmitted is processing
and so on. These resources are wasted in
> this period. Thus, we want to put TaskScheduler.start back to shorten cluster resources
occupation period specially for busy cluster.
> TaskScheduler could be started just before running stages.
> We could analyse and compare the  resources occupation period before and after optimization.
> TaskScheduler.start execution time: [time1__]
> DAGScheduler.JobSubmitted (excluding HadoopRDD.getPartitions or TaskScheduler.start)
execution time: [time2_]
> HadoopRDD.getPartitions execution time: [time3___]
> Stages execution time: [time4_____]
> The cluster resources occupation period before optimization is [time2_][time3___][time4_____].
> The cluster resources occupation period after optimization is....[time3___][time4_____].
> In summary, the cluster resources
> occupation period after optimization is less than before.
> If HadoopRDD.getPartitions could be put forward (SPARK-4961), the period may be shorten
more which is [time4_____].
> The resources saving is important for busy cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message