kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4367) MirrorMaker shuts down gracefully without actually being stopped
Date Wed, 09 Nov 2016 10:29:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15650572#comment-15650572
] 

Alex commented on KAFKA-4367:
-----------------------------

No, we didn't! That was the issue! Thanks for mentioning. We had to use nohup. 

> MirrorMaker shuts down gracefully without actually being stopped
> ----------------------------------------------------------------
>
>                 Key: KAFKA-4367
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4367
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 0.9.0.1
>         Environment: RHEL 7
>            Reporter: Alex
>
> MirrorMaker stops working without being stopped. From 30 minutes to 20 hours. No clue
why this problem occurs.
> Start:
> bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/ssl_mirroring_consumer.properties
--producer.config config/ssl_mirroring_producer.properties --whitelist "TOPIC1|TOPIC2|TOPIC3|TOPIC4"
--num.streams 20 &> /dev/null &
> --------------------------------------------------------------------------------
>           kafka-mirror-maker.log
> --------------------------------------------------------------------------------
> [2016-11-01 19:23:32,003] TRACE Produced messages to topic-partition CEP.FS.IN-175 with
base offset offset 15015 and error: null. (org.apache.kafka.clients.producer.internals.RecordBatch)
> [2016-11-01 19:23:32,003] TRACE Produced messages to topic-partition CEP.FS.IN-151 with
base offset offset 15066 and error: null. (org.apache.kafka.clients.producer.internals.RecordBatch)
> [2016-11-01 19:23:32,003] TRACE Nodes with data ready to send: [Node(8, 10.126.0.2, 9092)]
(org.apache.kafka.clients.producer.internals.Sender)
> [2016-11-01 19:23:32,003] TRACE Created 1 produce requests: [ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.producer.internals.Sender$1@483c4c7a, request=RequestSend(header={api_key=0,api_version=1,correlation_id=219685,client_id=producer-1},
body={acks=-1,timeout=30000,topic_data=[{topic=CEP.FS.IN,data=[{partition=133,record_set=java.nio.HeapByteBuffer[pos=0
lim=9085 cap=16384]}]}]}), createdTimeMs=1478017412003, sendTimeMs=0)] (org.apache.kafka.clients.producer.internals.Sender)
> [2016-11-01 19:23:32,008] TRACE Returning fetched records for assigned partition CEP.FS.IN-172
and update consumed position to 3869316 (org.apache.kafka.clients.consumer.internals.Fetcher)
> [2016-11-01 19:23:32,008] TRACE [mirrormaker-thread-7] Sending message with value size
485 and offset 3869315 (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,008] TRACE Sending record ProducerRecord(topic=CEP.FS.IN, partition=null,
key=null, value=[B@12a54f5a with callback kafka.tools.MirrorMaker$MirrorMakerProducerCallback@5ea65b8f
to topic CEP.FS.IN partition 160 (org.apache.kafka.clients.producer.KafkaProducer)
> [2016-11-01 19:23:32,008] TRACE Allocating a new 16384 byte message buffer for topic
CEP.FS.IN partition 160 (org.apache.kafka.clients.producer.internals.RecordAccumulator)
> [2016-11-01 19:23:32,008] TRACE Waking up the sender since topic CEP.FS.IN partition
160 is either full or getting a new batch (org.apache.kafka.clients.producer.KafkaProducer)
> [2016-11-01 19:23:32,010] TRACE Received produce response from node 7 with correlation
id 219684 (org.apache.kafka.clients.producer.internals.Sender)
> [2016-11-01 19:23:32,010] TRACE Produced messages to topic-partition CEP.FS.IN-106 with
base offset offset 15086 and error: null. (org.apache.kafka.clients.producer.internals.RecordBatch)
> [2016-11-01 19:23:32,010] TRACE Produced messages to topic-partition CEP.FS.IN-124 with
base offset offset 15095 and error: null. (org.apache.kafka.clients.producer.internals.RecordBatch)
> [2016-11-01 19:23:32,010] TRACE Nodes with data ready to send: [Node(7, 10.126.0.1, 9092)]
(org.apache.kafka.clients.producer.internals.Sender)
> [2016-11-01 19:23:32,010] INFO Start clean shutdown. (kafka.tools.MirrorMaker$)
> [2016-11-01 19:23:32,010] TRACE Created 1 produce requests: [ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.producer.internals.Sender$1@44b788c7, request=RequestSend(header={api_key=0,api_version=1,correlation_id=219686,client_id=producer-1},
body={acks=-1,timeout=30000,topic_data=[{topic=CEP.FS.IN,data=[{partition=160,record_set=java.nio.HeapByteBuffer[pos=0
lim=511 cap=16384]}]}]}), createdTimeMs=1478017412010, sendTimeMs=0)] (org.apache.kafka.clients.producer.internals.Sender)
> [2016-11-01 19:23:32,010] INFO Shutting down consumer threads. (kafka.tools.MirrorMaker$)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-0] mirrormaker-thread-0 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-1] mirrormaker-thread-1 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-2] mirrormaker-thread-2 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-3] mirrormaker-thread-3 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-4] mirrormaker-thread-4 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-5] mirrormaker-thread-5 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-6] mirrormaker-thread-6 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-7] mirrormaker-thread-7 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-8] mirrormaker-thread-8 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-9] mirrormaker-thread-9 shutting down
(kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-10] mirrormaker-thread-10 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-11] mirrormaker-thread-11 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-12] mirrormaker-thread-12 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-13] mirrormaker-thread-13 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-14] mirrormaker-thread-14 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-15] mirrormaker-thread-15 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-16] mirrormaker-thread-16 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-17] mirrormaker-thread-17 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-18] mirrormaker-thread-18 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-9] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-11] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-12] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-6] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-1] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] INFO [mirrormaker-thread-19] mirrormaker-thread-19 shutting
down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-13] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] INFO [mirrormaker-thread-13] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
> [2016-11-01 19:23:32,012] TRACE [mirrormaker-thread-4] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-19] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] INFO [mirrormaker-thread-19] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
> [2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-2] Caught ConsumerWakeupException,
continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] INFO [mirrormaker-thread-2] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
> --------------------------------------------------------------------------------
>       server.log
> --------------------------------------------------------------------------------
> [2016-11-01 19:23:32,031] INFO [GroupCoordinator 0]: Preparing to restabilize group MirrorringGroup
with old generation 2 (kafka.coordinator.GroupCoordinator)
> [2016-11-01 19:24:01,880] INFO [GroupCoordinator 0]: Group MirrorringGroup generation
2 is dead and removed (kafka.coordinator.GroupCoordinator)
> [2016-11-01 19:30:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired
offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
> [2016-11-01 19:40:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired
offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
> [2016-11-01 19:50:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired
offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message