flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebastian Klemke (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3188) Deletes in Kafka source should be passed on to KeyedDeserializationSchema
Date Fri, 18 Dec 2015 17:52:47 GMT
Sebastian Klemke created FLINK-3188:
---------------------------------------

             Summary: Deletes in Kafka source should be passed on to KeyedDeserializationSchema
                 Key: FLINK-3188
                 URL: https://issues.apache.org/jira/browse/FLINK-3188
             Project: Flink
          Issue Type: Bug
          Components: Kafka Connector
    Affects Versions: 1.0.0
            Reporter: Sebastian Klemke


When keys are deleted in the kafka queue, they show up as keys with null payload. Currently
in Flink 1.0-SNAPSHOT, these deletions are silently skipped, without increasing current offset.

This leads to two problems:

1. When a fetch window contains only deletions, LegacyFetcher gets stuck
2. For KeyedDeserializationSchemas, it would make sense to pass deletions to the deserializer,
so that it can decide to wrap deleted keys as a deletion command. This is also more consistent
with the semantics of keys in Kafka queues: When compaction is activated, only the latest
message with the same key needs to be kept by Kafka.

We propose the attached patch as a workaround for both issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message