beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Gerver (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-3525) TestPipeline serializes PipelineOptions prematurely
Date Wed, 24 Jan 2018 15:42:00 GMT
Paul Gerver created BEAM-3525:
---------------------------------

             Summary: TestPipeline serializes PipelineOptions prematurely
                 Key: BEAM-3525
                 URL: https://issues.apache.org/jira/browse/BEAM-3525
             Project: Beam
          Issue Type: Bug
          Components: testing
    Affects Versions: 2.2.0
            Reporter: Paul Gerver
            Assignee: Paul Gerver


The TestPipeline in its run() method now serializes PipelineOptions and adds in TestValueProvider
values before submitting the pipeline to run [1].


This premature serialization can cause options marked with JsonIgnore annotations to be dropped
before a runner has had a chance to see those parameters. For example, take the FlinkRunner
options, if flinkMaster is marked with JsonIgnore because that information is only needed
by the runner submitting the job and is not needed during runtime, tests will be run locally
incorrectly instead of against the Flink cluster.

[1] https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java#L354



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message