kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Attila Sasvari (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-6703) MirrorMaker cannot make progress when any matched topic from a whitelist regexp has -1 leader
Date Fri, 23 Mar 2018 16:10:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Attila Sasvari resolved KAFKA-6703.
-----------------------------------
    Resolution: Not A Bug

It turned out I forgot to set proper replication factor for {{__consumer_offsets}}. The default
replication factor is 1, and the consumer group controller (determined by {{partitionFor(group)}} in
GroupCoordinator) was down. Changing replication factor to 3, I did not experience the issue.
I still saw a couple of messages like
{code:java}
[2018-03-23 16:45:52,298] DEBUG [Consumer clientId=2-1, groupId=2] Leader for partition testR1P3-2
is unavailable for fetching offset (org.apache.kafka.clients.consumer.internals.Fetcher){code}
but messages in other topics matched by the whitelist regexp were fetched by MirrorMaker.

> MirrorMaker cannot make progress when any matched topic from a whitelist regexp has -1
leader
> ---------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6703
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6703
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 1.1.0
>            Reporter: Attila Sasvari
>            Priority: Major
>
> Scenario:
>  - MM whitelabel regexp matches multiple topics
>  - destination cluster has 5 brokers with multiple topics replication factor 3
>  - without partition reassign shut down 2 brokers
>  - suppose a topic has no leader any more because it was off-sync and the leader and
the rest of the replicas are hosted on the downed brokers.
>  - so we have 1 topic with some partitions with leader -1
>  - the rest of the matching topics has 3 replicas with leaders
> MM will not produce into any of the matched topics until:
>  - the "orphaned" topic removed or
>  - the partition reassign carried out from the downed brokers (suppose you can turn these
back on)
> In the MirrorMaker logs, there are a lot of messages like the following ones:
> {code}
> [2018-03-22 19:55:32,522] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054]
Coordinator discovery failed, refreshing metadata (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2018-03-22 19:55:32,522] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054]
Sending metadata request (type=MetadataRequest, topics=<ALL>) to node 192.168.1.102:9092
(id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
> [2018-03-22 19:55:32,525] DEBUG Updated cluster metadata version 10 to Cluster(id = Y-qtoFP-RMq2uuVnkEKAAw,
nodes = [192.168.1.102:9092 (id: 0 rack: null)], partitions = [Partition(topic = testR1P2,
partition = 1, leader = none, replicas = [42], isr = [], offlineReplicas = [42]), Partition(topic
= testR1P1, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas = []), Partition(topic
= testAlive, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas = []),
Partition(topic = testERRR, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas
= []), Partition(topic = testR1P2, partition = 0, leader = 0, replicas = [0], isr = [0], offlineReplicas
= [])]) (org.apache.kafka.clients.Metadata)
> [2018-03-22 19:55:32,525] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054]
Sending FindCoordinator request to broker 192.168.1.102:9092 (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2018-03-22 19:55:32,525] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054]
Received FindCoordinator response ClientResponse(receivedTimeMs=1521744932525, latencyMs=0,
disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=1, clientId=consumer-1,
correlationId=19), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null',
error=COORDINATOR_NOT_AVAILABLE, node=:-1 (id: -1 rack: null))) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2018-03-22 19:55:32,526] DEBUG [Consumer clientId=consumer-1, groupId=console-consumer-43054]
Group coordinator lookup failed: The coordinator is not available. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> {code}
> Interestingly, if MirrorMaker uses {{zookeeper.connect}} in its consumer properties file,
then an OldConsumer is created, and it can make progress.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message