spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng Lian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-975) Spark Replay Debugger
Date Mon, 18 Aug 2014 03:07:19 GMT

    [ https://issues.apache.org/jira/browse/SPARK-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100239#comment-14100239
] 

Cheng Lian commented on SPARK-975:
----------------------------------

Usually we just filter them out by checking package/class names . Similar trick is used in
{{org.apache.spark.util.Utils.getCallSite}}.

> Spark Replay Debugger
> ---------------------
>
>                 Key: SPARK-975
>                 URL: https://issues.apache.org/jira/browse/SPARK-975
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 0.9.0
>            Reporter: Cheng Lian
>              Labels: arthur, debugger
>         Attachments: IMG_20140722_184149.jpg, RDD DAG.png
>
>
> The Spark debugger was first mentioned as {{rddbg}} in the [RDD technical report|http://www.cs.berkeley.edu/~matei/papers/2011/tr_spark.pdf].
> [Arthur|https://github.com/mesos/spark/tree/arthur], authored by [Ankur Dave|https://github.com/ankurdave],
is an old implementation of the Spark debugger, which demonstrated both the elegance and power
behind the RDD abstraction.  Unfortunately, the corresponding GitHub branch was not merged
into the master branch and had stopped 2 years ago.  For more information about Arthur, please
refer to [the Spark Debugger Wiki page|https://github.com/mesos/spark/wiki/Spark-Debugger]
in the old GitHub repository.
> As a useful tool for Spark application debugging and analysis, it would be nice to have
a complete Spark debugger.  In [PR-224|https://github.com/apache/incubator-spark/pull/224],
I propose a new implementation of the Spark debugger, the Spark Replay Debugger (SRD).
> [PR-224|https://github.com/apache/incubator-spark/pull/224] is only a preview for discussion.
 In the current version, I only implemented features that can illustrate the basic mechanisms.
 There are still features appeared in Arthur but missing in SRD, such as checksum based nondeterminsm
detection and single task debugging with conventional debugger (like {{jdb}}).  However, these
features can be easily built upon current SRD framework.  To minimize code review effort,
I didn't include them into the current version intentionally.
> Attached is the visualization of the MLlib ALS application (with 1 iteration) generated
by SRD.  For more information, please refer to [the SRD overview document|http://spark-replay-debugger-overview.readthedocs.org/en/latest/].



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message