beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Gerver (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (BEAM-3525) TestPipeline serializes PipelineOptions prematurely
Date Thu, 25 Jan 2018 18:14:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Paul Gerver closed BEAM-3525.
-----------------------------
       Resolution: Workaround
    Fix Version/s: Not applicable

PipelineOptions need to be serialized for non-Java SDKs before they get to the runner. As
such, options that need to be present in the runner should not be marked with @JsonIgnore. 

Solution: Remove JsonIgnore annotation from runner configuration options.

> TestPipeline serializes PipelineOptions prematurely
> ---------------------------------------------------
>
>                 Key: BEAM-3525
>                 URL: https://issues.apache.org/jira/browse/BEAM-3525
>             Project: Beam
>          Issue Type: Bug
>          Components: testing
>    Affects Versions: 2.2.0
>            Reporter: Paul Gerver
>            Assignee: Paul Gerver
>            Priority: Minor
>             Fix For: Not applicable
>
>
> The TestPipeline in its run() method now serializes PipelineOptions and adds in TestValueProvider
values before submitting the pipeline to run [1].
> This premature serialization can cause options marked with JsonIgnore annotations to
be dropped before a runner has had a chance to see those parameters. For example, take the
FlinkRunner options, if flinkMaster is marked with JsonIgnore because that information is
only needed by the runner submitting the job and is not needed during runtime, tests will
be run locally incorrectly instead of against the Flink cluster.
> [1] https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java#L354



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message