kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aman Choudhary (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-3916) Connection from controller to broker disconnects
Date Sat, 26 Aug 2017 20:32:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142922#comment-16142922
] 

Aman Choudhary edited comment on KAFKA-3916 at 8/26/17 8:31 PM:
----------------------------------------------------------------

Hi,
 I am using Kafka version 0.10.0.2 in the production environment. I am facing issues in my
kafka broker and consumer machines which is very similar to issue described here.
Controller logs are very similar to the one described above:



{panel:title=My title}
|WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,010] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,289] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,289] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,050] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,050] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,060] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
{panel}




Consumer error log:

||Heading 1||Heading 2||
|2017-08-26 23:47:54,738 WARN [kafka-producer-network-thread | producer-8] o.a.k.c.p.i.Sender
[kafka-producer-network-thread | (consumer group)] Got error produce response with correlation
id 10322 on topic-partition topic-0, retrying (9 attempts left). Error: NETWORK_EXCEPTION|Col
A2|


I am having 5000 topics right now with a retention period of 1 hour. The maximum size of data
during peak load is 3-4 GB in a machine and I am having 6 kafka broker machines of 6 core
and 16 GB RAM.
Can someone please point out if there's something wrong in my approach? Do I need to update
to latest version?



was (Author: void.aman93):
Hi,
 I am using Kafka version 0.10.0.2 in the production environment. I am facing issues in my
kafka broker and consumer machines which is very similar to issue described here.
Controller logs are very similar to the one described above:

 
||Heading 1||Heading 2||
|WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,204] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
 WARN [2017-08-26 19:19:27,205] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-1,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,4,5],zk_version=0,replicas=[3,4,5]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,010] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-26 20:46:41,009] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2604144,partition=0,controller_epoch=34,leader=1,leader_epoch=0,isr=[1,2,3],zk_version=0,replicas=[1,2,3]}],live_leaders=[{id=1,host=host-4,port=9092}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]}],live_leaders=[{id=3,host=host-1,port=9092}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,288] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,289] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-26 23:44:28,289] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-3,partition=0,controller_epoch=34,leader=3,leader_epoch=0,isr=[3,1,2],zk_version=0,replicas=[3,1,2]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,050] [Controller-6-to-broker-6-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-6-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]}
to broker host-6:9092 (id: 6 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,050] [Controller-6-to-broker-2-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-2-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]}
to broker host-5:9092 (id: 2 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-4-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-4-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null}]}
to broker host-3:9092 (id: 4 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-3-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-3-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null},{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null}]}
to broker host-1:9092 (id: 3 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,060] [Controller-6-to-broker-5-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-5-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]}],live_leaders=[{id=2,host=host-5,port=9092}]}
to broker host-2:9092 (id: 5 rack: null). Reconnecting to broker.
 WARN [2017-08-27 00:10:59,059] [Controller-6-to-broker-1-send-thread][] kafka.controller.RequestSendThread
- [Controller-6-to-broker-1-send-thread], Controller 6 epoch 34 fails to send request {controller_id=6,controller_epoch=34,partition_states=[{topic=topic-2.2606546,partition=0,controller_epoch=34,leader=2,leader_epoch=0,isr=[2,5,6],zk_version=0,replicas=[2,5,6]},{topic=topic-4,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[],zk_version=0,replicas=[0]}],live_brokers=[{id=5,end_points=[{port=9092,host=host-2,security_protocol_type=0}],rack=null},{id=6,end_points=[{port=9092,host=host-6,security_protocol_type=0}],rack=null},{id=2,end_points=[{port=9092,host=host-5,security_protocol_type=0}],rack=null},{id=1,end_points=[{port=9092,host=host-4,security_protocol_type=0}],rack=null},{id=3,end_points=[{port=9092,host=host-1,security_protocol_type=0}],rack=null},{id=4,end_points=[{port=9092,host=host-3,security_protocol_type=0}],rack=null}]}
to broker host-4:9092 (id: 1 rack: null). Reconnecting to broker.
|Col A2|


Consumer error log:

||Heading 1||Heading 2||
|2017-08-26 23:47:54,738 WARN [kafka-producer-network-thread | producer-8] o.a.k.c.p.i.Sender
[kafka-producer-network-thread | (consumer group)] Got error produce response with correlation
id 10322 on topic-partition topic-0, retrying (9 attempts left). Error: NETWORK_EXCEPTION|Col
A2|


I am having 5000 topics right now with a retention period of 1 hour. The maximum size of data
during peak load is 3-4 GB in a machine and I am having 6 kafka broker machines of 6 core
and 16 GB RAM.
Can someone please point out if there's something wrong in my approach? Do I need to update
to latest version?


> Connection from controller to broker disconnects
> ------------------------------------------------
>
>                 Key: KAFKA-3916
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3916
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller
>    Affects Versions: 0.9.0.1
>            Reporter: Dave Powell
>            Assignee: Jason Gustafson
>             Fix For: 0.10.1.0
>
>
> We recently upgraded from 0.8.2.1 to 0.9.0.1. Since then, several times per day, the
controllers in our clusters have their connection to all brokers disconnected, and then successfully
reconnected a few hundred ms later. Each time this occurs we see a brief spike in our 99th
percentile produce and consume times, reaching several hundred ms.
> Here is an example of what we're seeing in the controller.log:
> {code}
> [2016-06-28 14:15:35,416] WARN [Controller-151-to-broker-160-send-thread], Controller
151 epoch 106 fails to send request {…} to broker Node(160, broker.160.hostname, 9092).
Reconnecting to broker. (kafka.controller.RequestSendThread)
> java.io.IOException: Connection to 160 was disconnected before the response was read
>         at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
>         at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
>         at scala.Option.foreach(Option.scala:236)
>         at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
>         at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
>         at kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:129)
>         at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
>         at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
>         at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:180)
>         at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:171)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ... one each for all brokers (including the controller) ...
>  [2016-06-28 14:15:35,721] INFO [Controller-151-to-broker-160-send-thread], Controller
151 connected to Node(160, broker.160.hostname, 9092) for sending state change requests (kafka.controller.RequestSendThread)
> … one each for all brokers (including the controller) ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


Mime
View raw message