kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3841) MirrorMaker topic renaming
Date Tue, 14 Jun 2016 22:59:30 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ning Zhang updated KAFKA-3841:
------------------------------
    Description: 
Our organization (walmart.com) has been a Kafka user since some years back and MirrorMaker
has been a convenient tool to bring our Kafka data from one Kafka cluster to another cluster.

In many our use cases, the mirrored topic from the source cluster may not want to have the
same name in the target cluster. This could be a valid scenario when the same topic name already
exists on the target cluster, or we want to append the name of the data center to the topic
name in the target cluster, such as "grocery_items_mirror_sunnyvale", to explicitly identify
the source (e.g. sunnyvale) and nature (e.g. mirroring) of the topic.

We have implemented the MirrorMaker topic renaming feature internally which has been used
for production over a couple of years. While keeping our internal Kafka fork with the above
"renaming" branch across version upgrade does not cost us too much labor, we think it may
be meaningful to contribute back to the community so that potentially many people may have
the similar expectation and could benefit from this feature.


  was:
Our organization (walmart.com) has been a Kafka user since 3 years back and MirrorMaker has
been a convenient tool to bring our Kafka data from one Kafka cluster to another cluster.

In many our use cases, the mirrored topic from the source cluster may not want to have the
same name in the target cluster. This could be a valid scenario when the same topic name already
exists on the target cluster, or we want to append the name of the data center to the topic
name in the target cluster, such as "grocery_items_mirror_sunnyvale", to explicitly identify
the source (e.g. sunnyvale) and nature (e.g. mirroring) of the topic.

We have implemented the MirrorMaker topic renaming feature internally which has been used
for production over 2 years. Keeping our internal Kafka fork with the above "renaming" branch
across version upgrade does not cost us too much labor, but we think it may be meaningful
to contribute back to the community so that potentially many people may have the similar expectation
and could benefit from this feature.



> MirrorMaker topic renaming
> --------------------------
>
>                 Key: KAFKA-3841
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3841
>             Project: Kafka
>          Issue Type: New Feature
>          Components: tools
>    Affects Versions: 0.10.0.0
>            Reporter: Ning Zhang
>
> Our organization (walmart.com) has been a Kafka user since some years back and MirrorMaker
has been a convenient tool to bring our Kafka data from one Kafka cluster to another cluster.
> In many our use cases, the mirrored topic from the source cluster may not want to have
the same name in the target cluster. This could be a valid scenario when the same topic name
already exists on the target cluster, or we want to append the name of the data center to
the topic name in the target cluster, such as "grocery_items_mirror_sunnyvale", to explicitly
identify the source (e.g. sunnyvale) and nature (e.g. mirroring) of the topic.
> We have implemented the MirrorMaker topic renaming feature internally which has been
used for production over a couple of years. While keeping our internal Kafka fork with the
above "renaming" branch across version upgrade does not cost us too much labor, we think it
may be meaningful to contribute back to the community so that potentially many people may
have the similar expectation and could benefit from this feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message