flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3435) Change interplay of Ingestion Time and Event Time
Date Wed, 17 Feb 2016 15:55:18 GMT
Stephan Ewen created FLINK-3435:
-----------------------------------

             Summary: Change interplay of Ingestion Time and Event Time
                 Key: FLINK-3435
                 URL: https://issues.apache.org/jira/browse/FLINK-3435
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
    Affects Versions: 0.10.2
            Reporter: Stephan Ewen
             Fix For: 1.0.0


Currently, "EventTime" and "IngestionTime" are completely the same.

For both happens the following:
  - Sources generate ingestion time timestamps and watermarks
  - If a user adds a manual timestamp extractor / watermark generator, then those override
the ingestion time timestamps and watermarks
  - That implies that event time on a certain input falls back to ingestion time, if one forgets
(or incorrectly uses) the timestamp extractors
  - Also, Ingestion Time and Event Time simply mix if some inputs have timestamp assigners,
and others have not.

This behavior is quite tricky to understand. After some discussions with [~aljoscha] and [~rmetzger],
we suggest to change it the following way.

  1. On Ingestion Time, the timestamps and watermarks are generated as they are now.

  2. On event time, no default timestamps and watermarks are generated. If a user does not
implement a timestamp extractor / watermark generator, then the event time operations will
fail fast.

  3. If one wants to use ingestion time on event time settings (mix), one can use an explicit
"WallClockTimetampsAndWatermark" generator.

  4. Later, the "Ingestion Time" settings should automatically disable and user-defined timestamp
extractors / assigners.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message