spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Yuanjian (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark
Date Wed, 06 Jun 2018 15:56:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503486#comment-16503486
] 

Li Yuanjian edited comment on SPARK-24375 at 6/6/18 3:55 PM:
-------------------------------------------------------------

Hi [~cloud_fan] and [~jiangxb1987], just I tiny question here, I notice the discussion in
SPARK-20928 mentioned the barrier scheduling which Continuous Processing will depend on.
{quote}
A barrier stage doesn’t launch any of its tasks until the available slots(free CPU cores
can be used to launch pending tasks) satisfies the target to launch all the tasks at the same
time, and always retry the whole stage when any task(s) fail.
{quote}
Why the task level retrying was forbidden here, is there any possible to achieve this? Thanks.
 

 


was (Author: xuanyuan):
Hi [~cloud_fan] and [~jiangxb1987], just I tiny question here, I notice the discussion in
SPARK-20928 mentioned the barrier scheduling.
{quote}
A barrier stage doesn’t launch any of its tasks until the available slots(free CPU cores
can be used to launch pending tasks) satisfies the target to launch all the tasks at the same
time, and always retry the whole stage when any task(s) fail.
{quote}
Why the task level retrying was forbidden here, is there any possible to achieve this? Thanks.
 

 

> Design sketch: support barrier scheduling in Apache Spark
> ---------------------------------------------------------
>
>                 Key: SPARK-24375
>                 URL: https://issues.apache.org/jira/browse/SPARK-24375
>             Project: Spark
>          Issue Type: Story
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: Xiangrui Meng
>            Assignee: Jiang Xingbo
>            Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP discussion. It
doesn't need to be a complete design before the vote. But it should at least cover both Scala/Java
and PySpark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message