spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-11361) Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG viz
Date Wed, 11 Nov 2015 00:54:11 GMT

     [ https://issues.apache.org/jira/browse/SPARK-11361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or resolved SPARK-11361.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 1.6.0

> Show scopes of RDD operations inside DStream.foreachRDD and DStream.transform in DAG
viz
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-11361
>                 URL: https://issues.apache.org/jira/browse/SPARK-11361
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>            Priority: Minor
>             Fix For: 1.6.0
>
>
> Currently, when a DStream sets the scope for RDD generated by it, that scope is not allowed
to be overridden by the RDD operations. So in case of `DStream.foreachRDD`, all the RDDs generated
inside the foreachRDD get the same scope - `foreachRDD @ <time>`, as set by the `ForeachDStream`.
So it is hard to debug generated RDDs in the RDD DAG viz in the Spark UI. 
> This JIRA is to allow the RDD operations inside `DStream.transform` and `DStream.foreachRDD`
to append their own scopes to the earlier DStream scope. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message