kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5567) With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition
Date Fri, 07 Jul 2017 05:17:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077603#comment-16077603
] 

ASF GitHub Bot commented on KAFKA-5567:
---------------------------------------

GitHub user kkonstantine opened a pull request:

    https://github.com/apache/kafka/pull/3499

    KAFKA-5567: Connect sink worker should commit offsets of original topic partitions

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kkonstantine/kafka KAFKA-5567-With-transformations-that-mutate-the-topic-partition-committing-offsets-should-to-refer-to-the-original-topic-partition

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/3499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3499
    
----
commit b9eeeb12812f0b237b4356717dfc5bf0e252e8e6
Author: Konstantine Karantasis <konstantine@confluent.io>
Date:   2017-07-07T05:13:50Z

    KAFKA-5567: Connect sink worker should commit offsets of original topic partitions.

----


> With transformations that mutate the topic-partition committing offsets should to refer
to the original topic-partition
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5567
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5567
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 0.10.2.0, 0.10.2.1, 0.11.0.0
>            Reporter: Konstantine Karantasis
>            Assignee: Konstantine Karantasis
>             Fix For: 0.11.0.1
>
>
>   When a chain of transformations (SMTs) that mutate a record's topic-partition is applied,
then Connect is unable to map the transformed record to its original topic-partition. This
affects committing offsets. 
>  Currently, in order to reproduce the issue one could use the {{TimestampRouter}} transformation
with a sink connector such as the {{FileStreamSinkConnector}}.
>   In this ticket we'll address the issue for connectors that don't manage/commit their
offsets themselves. For the connectors that do such management, broader API changes are required
to supply the connectors with the necessary information that will allow them to map a transformed
record to the original. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message