kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Fung (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-599) SimpleConsumerShell ONLY connects to the first host in the broker-list string to fetch topic metadata
Date Fri, 02 Nov 2012 17:07:12 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489541#comment-13489541
] 

John Fung edited comment on KAFKA-599 at 11/2/12 5:05 PM:
----------------------------------------------------------

** This is showing in testcase_0253 in which the first broker (out of the 3-brokers cluster)
is down. SimpleConsumerShell doesn't seem to try to connect to other available brokers to
fetch topic metadata.


1. System Test output showing the command line arguments of SimpleConsumerShell

2012-11-02 06:52:15,728 - INFO - starting debug consumer for replica on [host_0997:9091] partition
[0] (kafka_system_test_utils)
2012-11-02 06:52:15,729 - DEBUG - executing command: [ssh host_0996.prod 'JAVA_HOME=/export/apps/jdk/JDK-1_6_0_21
/kafka_08_replication_system_test/bin/kafka-run-class.sh kafka.tools.SimpleConsumerShell --broker-list
host_0997:9091,host_0998:9092,host_0999:9093 --topic test_1 --partition 0 --replica 1 --offset
-2 --no-wait-at-logend   >> /mnt/u001/kafka_08_replication_system_test/system_test/replication_testsuite/testcase_0253/logs/console_consumer-5/simple_consumer_1.log
 & echo pid:$! > /mnt/u001/kafka_08_replication_system_test/system_test/replication_testsuite/testcase_0253/logs/console_consumer-5/entity_5_pid']
(kafka_system_test_utils)

2. SimpleConsumerShell keeps trying on the first host to fetch topic metadata:

[2012-11-02 06:52:16,290] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
[2012-11-02 06:52:16,352] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:16,361] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:16,364] WARN fetching topic metadata for topics [Set(test_1)] from broker
[id:0,creatorId:host_0997-1351839136304,host:host_0997,port:9091] failed (kafka.client.ClientUtils$)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:16,364] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:16,366] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer)
[2012-11-02 06:52:16,412] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer)
[2012-11-02 06:52:18,324] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
[2012-11-02 06:52:18,386] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:18,395] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:18,398] WARN fetching topic metadata for topics [Set(test_1)] from broker
[id:0,creatorId:host_0997-1351839138338,host:host_0997,port:9091] failed (kafka.client.ClientUtils$)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:18,399] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:18,401] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer)
[2012-11-02 06:52:18,446] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer)
[2012-11-02 06:52:20,355] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
[2012-11-02 06:52:20,417] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:20,426] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)


                
      was (Author: jfung):
    ** This is showing in testcase_0253 in which the first broker (out of the 3-brokers cluster)
is down. SimpleConsumerShell doesn't seem to try to connect to other available brokers to
fetch topic metadata.


1. System Test output showing the command line arguments of SimpleConsumerShell

2012-11-02 06:52:15,728 - INFO - starting debug consumer for replica on [host_0997:9091] partition
[0] (kafka_system_test_utils)
2012-11-02 06:52:15,729 - DEBUG - executing command: [ssh host_0996.prod 'JAVA_HOME=/export/apps/jdk/JDK-1_6_0_21
/mnt/u001/kafka_08_replication_system_test/bin/kafka-run-class.sh kafka.tools.SimpleConsumerShell
--broker-list host_0997:9091,host_0998:9092,host_0999:9093 --topic test_1 --partition 0 --replica
1 --offset -2 --no-wait-at-logend   >> /mnt/u001/kafka_08_replication_system_test/system_test/replication_testsuite/testcase_0253/logs/console_consumer-5/simple_consumer_1.log
 & echo pid:$! > /mnt/u001/kafka_08_replication_system_test/system_test/replication_testsuite/testcase_0253/logs/console_consumer-5/entity_5_pid']
(kafka_system_test_utils)

2. SimpleConsumerShell keeps trying on the first host to fetch topic metadata:

[2012-11-02 06:52:16,290] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
[2012-11-02 06:52:16,352] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:16,361] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:16,364] WARN fetching topic metadata for topics [Set(test_1)] from broker
[id:0,creatorId:host_0997-1351839136304,host:host_0997,port:9091] failed (kafka.client.ClientUtils$)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:16,364] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:16,366] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer)
[2012-11-02 06:52:16,412] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer)
[2012-11-02 06:52:18,324] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
[2012-11-02 06:52:18,386] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:18,395] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:18,398] WARN fetching topic metadata for topics [Set(test_1)] from broker
[id:0,creatorId:host_0997-1351839138338,host:host_0997,port:9091] failed (kafka.client.ClientUtils$)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at kafka.producer.SyncProducer.connect(SyncProducer.scala:135)
        at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:150)
        at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
        at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:25)
        at kafka.tools.SimpleConsumerShell$.main(SimpleConsumerShell.scala:128)
        at kafka.tools.SimpleConsumerShell.main(SimpleConsumerShell.scala)
[2012-11-02 06:52:18,399] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:18,401] INFO Connected to host_0998:9092 for producing (kafka.producer.SyncProducer)
[2012-11-02 06:52:18,446] INFO Disconnecting from host_0998:9092 (kafka.producer.SyncProducer)
[2012-11-02 06:52:20,355] INFO Getting topic metatdata... (kafka.tools.SimpleConsumerShell$)
[2012-11-02 06:52:20,417] INFO Fetching metadata for topic Set(test_1) (kafka.client.ClientUtils$)
[2012-11-02 06:52:20,426] ERROR Producer connection to host_0997:9091 unsuccessful (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)


                  
> SimpleConsumerShell ONLY connects to the first host in the broker-list string to fetch
topic metadata
> -----------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-599
>                 URL: https://issues.apache.org/jira/browse/KAFKA-599
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: John Fung
>            Priority: Critical
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message