beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chamikara Jayalath (JIRA)" <>
Subject [jira] [Created] (BEAM-738) Update sourcetestutils to fail when BoundedSources maintain state
Date Mon, 10 Oct 2016 21:13:20 GMT
Chamikara Jayalath created BEAM-738:

             Summary: Update sourcetestutils to fail when BoundedSources maintain state
                 Key: BEAM-738
             Project: Beam
          Issue Type: Improvement
          Components: sdk-py
            Reporter: Chamikara Jayalath
            Assignee: Chamikara Jayalath

Maintaining state in BoundedSource implementations is problematic and can lead to hard to
debug errors. For example (1) pickling errors (2) errors due to a runner reusing a BoundedSource
object with state.

We can try to prevent users from adding state to BoundedSource implementations in following
two ways.

(1) Clearly mention in BoundedSource API that objects should not maintain transient state.
(2) Update sourcetesutils to catch source objects that maintain local state.

(2) can be done by adding a check that verifies that a source produces expected output in
the presence of a re-entrant read.

i = s.read_records()
read the whole thing from s.read_records() some more

Verify that 'i' produced correct output.

This message was sent by Atlassian JIRA

View raw message