kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (KAFKA-6048) Support negative record timestamps
Date Tue, 28 Nov 2017 16:56:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Matthias J. Sax reassigned KAFKA-6048:

    Assignee: Konstantin Chukhlomin  (was: james chien)

> Support negative record timestamps
> ----------------------------------
>                 Key: KAFKA-6048
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6048
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, core, streams
>    Affects Versions: 1.0.0
>            Reporter: Matthias J. Sax
>            Assignee: Konstantin Chukhlomin
>              Labels: needs-kip
> Kafka does not support negative record timestamps, and this prevents the storage of historical
data in Kafka. In general, negative timestamps are supported by UNIX system time stamps: 
> From https://en.wikipedia.org/wiki/Unix_time
> {quote}
> The Unix time number is zero at the Unix epoch, and increases by exactly 86,400 per day
since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after the epoch, is represented by
the Unix time number 12,677 × 86,400 = 1095292800. This can be extended backwards from the
epoch too, using negative numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch,
is represented by the Unix time number −4,472 × 86,400 = −386380800.
> {quote}
> Allowing for negative timestamps would require multiple changes:
>  - while brokers in general do support negative timestamps, broker use {{-1}} as default
value if a producer uses an old message format (this would not be compatible with supporting
negative timestamps "end-to-end" as {{-1}} cannot be used as "unknown" anymore): we could
introduce a message flag indicating a missing timestamp (and let producer throw an exception
if {{ConsumerRecord#timestamp()}} is called. Another possible solution might be, to require
topics that are used by old producers to be configured with {{LogAppendTime}} semantics and
rejecting writes to topics with {{CreateTime}} semantics for older message formats
>  - {{KafkaProducer}} does not allow to send records with negative timestamp and thus
this would need to be fixed
>  - Streams API does drop records with negative timestamps (or fails by default) -- also,
some internal store implementation for windowed stores assume that there are not negative
timestamps to do range queries
> There might be other gaps we need to address. This is just a summary of issue coming
to my mind atm.

This message was sent by Atlassian JIRA

View raw message