beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (BEAM-797) A PipelineVisitor that creates a Spark-native pipeline.
Date Fri, 10 Mar 2017 13:37:04 GMT

     [ https://issues.apache.org/jira/browse/BEAM-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Amit Sela resolved BEAM-797.
----------------------------
       Resolution: Fixed
    Fix Version/s: First stable release

> A PipelineVisitor that creates a Spark-native pipeline. 
> --------------------------------------------------------
>
>                 Key: BEAM-797
>                 URL: https://issues.apache.org/jira/browse/BEAM-797
>             Project: Beam
>          Issue Type: Wish
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Aviem Zur
>            Priority: Minor
>             Fix For: First stable release
>
>
> It could be very useful for debugging purposes to have a custom PipelineVisitor that
can tell what's the underlying Spark code that is being called.
> One idea:
> This could be called with a flag in SparkPipelineOptions and instead of executing the
pipeline, it would print the underlying Spark DAG.
> Clearly, DoFn internals would be obfuscated, but the Spark code could note {{mapPartitions("ExtractWords")}}
> Another difference would be Sources as they are a custom implementation for Beam.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message