beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Charles Chen (JIRA)" <>
Subject [jira] [Created] (BEAM-3512) Python PTransform overrides do not completely remove the overriden transform
Date Mon, 22 Jan 2018 20:31:00 GMT
Charles Chen created BEAM-3512:

             Summary: Python PTransform overrides do not completely remove the overriden transform
                 Key: BEAM-3512
             Project: Beam
          Issue Type: Bug
          Components: sdk-py-core
    Affects Versions: 2.2.0
            Reporter: Charles Chen
            Assignee: Charles Chen

In Python Beam runners, we support the use of PTransformOverrides to allow runners to override
the behavior of specific transforms.  The overriding mechanism seeks to excise the original
transform (with its composite children transforms, if any) from the original graph, and graft
the new replacement to any inputs and outputs of the original transform.  However, this mechanism
does not completely remove these pieces.  Specifically:

1. Composite transform parts are not removed from the overridden AppliedPTransform; we only
attempt to remove all the labels.

2. Not all labels are recursively removed; there is a bug in the label removing logic so that
only the labels on every other level of the nested composite PTransform hierarchy are removed
(see []).

3. Some excised parts of the pipeline still seem to be run: if we set "part.transform = None"
for each overridden composite part of the pipeline, some tests fail with an error indicating
that it could not execute the "None" transform.  This implies that these parts are still
somehow present in the execution graph.

This message was sent by Atlassian JIRA

View raw message