spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-1015) Visualize the DAG of RDD
Date Thu, 26 Feb 2015 11:08:04 GMT


Sean Owen commented on SPARK-1015:

[~zjffdu] are you planning on working on this? We also have {{toDebugString}} which prints
some of this info. How would the visualization work with spark-shell? Is this just a utility
you can host outside Spark?

> Visualize the DAG of RDD 
> -------------------------
>                 Key: SPARK-1015
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 0.9.0
>            Reporter: Jeff Zhang
> The DAG of RDD can help user understand the data flow and how spark get the final RDD
executed.  It could help user to find chances to optimize the execution of some complex RDD.
 I will leverage graphviz to visualize the DAG. 
> For this task, I plan to split it into 2 steps.
> Step 1.  Just visualize the simple DAG graph.  Each RDD is one node, and there will be
one edge between the parent RDD and child RDD. ( I attach one simple graph in the attachments
> Step 2.  Put RDD in the same stage into one sub graph. This may need to extract the splitting
staging related code in DAGSchduler. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message