spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mridul Muralidharan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage
Date Wed, 03 May 2017 20:07:04 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995550#comment-15995550
] 

Mridul Muralidharan commented on SPARK-20589:
---------------------------------------------

coalasce with shuffle=false might be a workaround if source is already persisted ?
(I see the benefit of the jira, just wondering if this will unblock you !)

> Allow limiting task concurrency per stage
> -----------------------------------------
>
>                 Key: SPARK-20589
>                 URL: https://issues.apache.org/jira/browse/SPARK-20589
>             Project: Spark
>          Issue Type: Improvement
>          Components: Scheduler
>    Affects Versions: 2.1.0
>            Reporter: Thomas Graves
>
> It would be nice to have the ability to limit the number of concurrent tasks per stage.
 This is useful when your spark job might be accessing another service and you don't want
to DOS that service.  For instance Spark writing to hbase or Spark doing http puts on a service.
 Many times you want to do this without limiting the number of partitions. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message