beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <>
Subject [jira] [Resolved] (BEAM-797) A PipelineVisitor that creates a Spark-native pipeline.
Date Fri, 10 Mar 2017 13:37:04 GMT


Amit Sela resolved BEAM-797.
       Resolution: Fixed
    Fix Version/s: First stable release

> A PipelineVisitor that creates a Spark-native pipeline. 
> --------------------------------------------------------
>                 Key: BEAM-797
>                 URL:
>             Project: Beam
>          Issue Type: Wish
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Aviem Zur
>            Priority: Minor
>             Fix For: First stable release
> It could be very useful for debugging purposes to have a custom PipelineVisitor that
can tell what's the underlying Spark code that is being called.
> One idea:
> This could be called with a flag in SparkPipelineOptions and instead of executing the
pipeline, it would print the underlying Spark DAG.
> Clearly, DoFn internals would be obfuscated, but the Spark code could note {{mapPartitions("ExtractWords")}}
> Another difference would be Sources as they are a custom implementation for Beam.

This message was sent by Atlassian JIRA

View raw message