kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Wilkinson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-658) Implement "Exact Mirroring" functionality in mirror maker
Date Mon, 14 Aug 2017 15:42:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125847#comment-16125847

Paul Wilkinson commented on KAFKA-658:

Does the recent work in exactly-once semantics mean this JIRA is likely to make progress?

> Implement "Exact Mirroring" functionality in mirror maker
> ---------------------------------------------------------
>                 Key: KAFKA-658
>                 URL: https://issues.apache.org/jira/browse/KAFKA-658
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Jay Kreps
>              Labels: project
> There are two ways to implement "mirroring" (i.e. replicating a topic from one cluster
to another):
> 1. Do a simple read from the source and write to the destination with no attempt to maintain
the same partitioning or offsets in the destination cluster. In this case the destination
cluster may have a different number of partitions, and you can even read from many clusters
to create a merged cluster. This flexibility is nice. The downside is that since the partitioning
and offsets are not the same a consumer of the source cluster has no equivalent position in
the destination cluster. This is the style of mirroring we have implemented in the mirror-maker
tool and use for datacenter replication today.
> 2. The second style of replication only would allow creating an exact replica of a source
cluster (i.e. all partitions and offsets exactly the same). The nice thing about this is that
the offsets and partitions would match exactly. The downside is that it is not possible to
merge multiple source clusters this way or have different partitioning. We do not currently
support this in mirror maker.
> It would be nice to implement the second style as an option in mirror maker as having
an exact replica would be a nice option to have in the case where you are replicating a single
cluster only.
> There are some nuances: In order to maintain the exact offsets it is important to guarantee
that the producer never resends a message or loses a message. As a result it would be important
to have only a single producer for each destination partition, and check the last produced
message on startup (using the getOffsets api) so that in the case of a hard crash messages
that are re-consumed are not re-emitted.

This message was sent by Atlassian JIRA

View raw message