beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Pettitt (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow
Date Mon, 31 Jul 2017 20:22:00 GMT
Chris Pettitt created BEAM-2703:
-----------------------------------

             Summary: KafkaIO: watermark outside the bounds of BoundedWindow
                 Key: BEAM-2703
                 URL: https://issues.apache.org/jira/browse/BEAM-2703
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
            Reporter: Chris Pettitt
            Assignee: Davor Bonaci


KafkaIO appears to use an incorrect lower bound for it's initial watermark with respect to
BoundedWindow.TIMESTAMP_MIN_VALUE.

KafkaIO's initial watermark:
new Instant(Long.MIN_VALUE) -> -9223372036854775808

BoundedWindow.TIMESTAMP_MIN_VALUE:
new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> -9223372036854775

The difference is that the last three digits have been truncated due to the micro to millis
conversion.

This difference can cause errors in runners that assert that the input watermark can never
regress as KafkaIO gives a value below the lower bound when no messages have been received
yet. For consistency it would probably be best for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message