flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3188) Deletes in Kafka source should be passed on to KeyedDeserializationSchema
Date Mon, 04 Jan 2016 11:10:39 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081011#comment-15081011

Robert Metzger commented on FLINK-3188:

Thank you for providing a test case. I'll open a pull request for the changes and merge them

> Deletes in Kafka source should be passed on to KeyedDeserializationSchema
> -------------------------------------------------------------------------
>                 Key: FLINK-3188
>                 URL: https://issues.apache.org/jira/browse/FLINK-3188
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>    Affects Versions: 1.0.0
>            Reporter: Sebastian Klemke
>            Assignee: Robert Metzger
>         Attachments: kafka-deletions.patch, testcase.patch
> When keys are deleted in the kafka queue, they show up as keys with null payload. Currently
in Flink 1.0-SNAPSHOT, these deletions are silently skipped, without increasing current offset.
> This leads to two problems:
> 1. When a fetch window contains only deletions, LegacyFetcher gets stuck
> 2. For KeyedDeserializationSchemas, it would make sense to pass deletions to the deserializer,
so that it can decide to wrap deleted keys as a deletion command. This is also more consistent
with the semantics of keys in Kafka queues: When compaction is activated, only the latest
message with the same key needs to be kept by Kafka.
> We propose the attached patch as a workaround for both issues.

This message was sent by Atlassian JIRA

View raw message