beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stas Levin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-2072) MicrobatchSource.reader stops reading after reaching maxNumRecords for the first time
Date Tue, 25 Apr 2017 08:57:04 GMT
Stas Levin created BEAM-2072:
--------------------------------

             Summary: MicrobatchSource.reader stops reading after reaching maxNumRecords for
the first time
                 Key: BEAM-2072
                 URL: https://issues.apache.org/jira/browse/BEAM-2072
             Project: Beam
          Issue Type: Bug
          Components: runner-spark
    Affects Versions: Not applicable
            Reporter: Stas Levin
            Assignee: Stas Levin


{{MicrobatchSource.Reader}} stops reading further data after the first batch in which it has
reached {{maxNumRecords}}. Subsequent read request will check the {{recordsRead}} and conclude
that no data can be read.

The root cause being that {{recordsRead}} is maintained throughout the reader's lifecycle,
and can span multiple (spark) batches. 
We should reset {{recordsRead}} every time the reader starts reading a fresh batch.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message