flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3429) Add histogram-based timestamp/watermark extractor
Date Wed, 17 Feb 2016 11:08:18 GMT
Robert Metzger created FLINK-3429:

             Summary: Add histogram-based timestamp/watermark extractor
                 Key: FLINK-3429
                 URL: https://issues.apache.org/jira/browse/FLINK-3429
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
            Reporter: Robert Metzger

While FLINK-3428 provides a static low watermark generator, we should also provide a watermark
extractor which periodically takes samples of the event-time distribution in a data stream
into a histogram.
>From that histogram, users can specify that watermarks should cover a certain percentile
of the data.

For example a user might want watermarks to rise when 90% of the events have been seen. With
the periodic update of the histogram, the extractor can react to network latencies and other
issues causing late, out-of-order arrivals.

This message was sent by Atlassian JIRA

View raw message