spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-10649) Streaming jobs unexpectedly inherits job group, job descriptions from context starting thread
Date Wed, 16 Sep 2015 20:55:46 GMT

     [ https://issues.apache.org/jira/browse/SPARK-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-10649:
------------------------------------

    Assignee: Tathagata Das  (was: Apache Spark)

> Streaming jobs unexpectedly inherits job group, job descriptions from context starting
thread
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-10649
>                 URL: https://issues.apache.org/jira/browse/SPARK-10649
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.3.1, 1.4.1, 1.5.0
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>
> The job group, job descriptions and scheduler pool information is passed through thread
local properties, and get inherited by child threads. In case of spark streaming, the streaming
jobs inherit these properties from the thread that called streamingContext.start(). This may
not make sense. 
> 1. Job group: This is mainly used for cancelling a group of jobs together. It does not
make sense to cancel streaming jobs like this, as the effect will be unpredictable. And its
not a valid usecase any way, to cancel a streaming context, call streamingContext.stop()
> 2. Job description: This is used to pass on nice text descriptions for jobs to show up
in the UI. The job description of the thread that calls streamingContext.start() is not useful
for all the streaming jobs, as it does not make sense for all of the streaming jobs to have
the same description, and the description may or may not be related to streaming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message