kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Randall Hauch (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-3821) Allow Kafka Connect source tasks to produce offset without writing to topics
Date Tue, 18 Apr 2017 21:17:41 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973506#comment-15973506
] 

Randall Hauch edited comment on KAFKA-3821 at 4/18/17 9:17 PM:
---------------------------------------------------------------

[~criccomini] logged KAFKA-5084, but that is really another situation where the connector
would like to provide new offsets without having to produce {{SourceRecord}}. As he says [here|https://confluentcommunity.slack.com/archives/C49L0V3L7/p1492547735471505],
the connector sometimes need to tell Kafka Connect:

{quote}
i made some progress in my source, but i don’t want to send any of the data
{quote}

This becomes an issue when stopping and restarting the connector. In cases when it's been
a long time since the connector produced source records with an updated offset, the connector
upon startup will receive this old (but still the latest) offset. But the source has long
purged that portion of the log, so the connector is unable to start up. In reality, the connector
had processed much farther than was described by the last offset, and if it had been able
to submit that offset to Kafka Connect, upon restart it would have been able to successfully
start at a point in the log that still exists in the source.


was (Author: rhauch):
[~criccomini] logged KAFKA-5084, but that is really another situation where the connector
would like to provide new offsets without having to produce {{SourceRecord}}. As he says [here|https://confluentcommunity.slack.com/archives/C49L0V3L7/p1492547735471505],
the connector sometimes need to tell Kafka Connect:

{quote}
i made some progress in my source, but i don’t want to send any of the data
{quote}

> Allow Kafka Connect source tasks to produce offset without writing to topics
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-3821
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3821
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.9.0.1
>            Reporter: Randall Hauch
>              Labels: needs-kip
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for a given
partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown number
of {{SourceRecord}} objects (e.g., it is taking a snapshot of a database). Once the task completes
those records, the connector wants to update the offsets (e.g., the snapshot is complete)
but has no more records to be written to a topic. With this change, the task could simply
supply an updated offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message