kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikumar (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-1497) Change producer load-balancing algorithm in MirrorMaker
Date Thu, 24 Aug 2017 14:29:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Manikumar resolved KAFKA-1497.
------------------------------
    Resolution: Fixed

MIrrorMaker now uses single producer instance.

> Change producer load-balancing algorithm in MirrorMaker
> -------------------------------------------------------
>
>                 Key: KAFKA-1497
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1497
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.8.1.1
>            Reporter: Ivan Kunz
>
> Currently the MirrorMaker uses the following way of spreading the load into configured
producers :
>             val producerId = Utils.abs(java.util.Arrays.hashCode(msgAndMetadata.key))
% producers.size()
> This way if the producer side of MM uses different than the default "partitioner.class"
messages within the same partition can get re-ordered. Also hashCode does not produce the
same results on different machines (verified by testing) so cannot be safely used for partitioning
between distributed systems connected via MM (for us message order preservation within a partition
is a critical feature).
> It would be great if the code above is changed to utilize the configured "partitioner.class".

> Something along the lines of  :
> At the initialization:
>       mmpartitioner = Utils.createObject[Partitioner](config.partitionerClass, config.props)
     
> During the processing:
>             val producerId = mmpartitioner.partition(msgAndMetadata.key,producers.size())
> This way the messages consumed and produced by MM can remain in the same order.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message