kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jun Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-369) remove ZK dependency on producer
Date Wed, 08 Aug 2012 17:03:21 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431220#comment-13431220
] 

Jun Rao commented on KAFKA-369:
-------------------------------

Thanks for patch v5. A few more comments:

50. There are a bunch of places where zk.connect is still used in ProducerConfig, could you
fix them?
ProducerPerformance, ReplayLogProducer, ConsoleProducer, TestEndToEndLatency, AsyncProducerTest,
KafkaOutputFormat.java, Producer.java

51. Single host multi broker system test seems to hang:
2012-08-08 09:08:22 =======================================
2012-08-08 09:08:22 Iteration 1 of 3
2012-08-08 09:08:22 =======================================

2012-08-08 09:08:22 looking up leader
2012-08-08 09:08:22 found the log line: [2012-08-08 09:08:17,434] INFO Controller 1, at initializing
leaders for new partitions, the leaderAndISR request sent to broker 3 is LeaderAndISRRequest(1,,false,1000,Map((mytest,0)
-> { "ISR": "1,2,3","leader": "1","leaderEpoch": "0" })) (kafka.server.KafkaController)
bin/run-test.sh: line 151: return: [2012-08-08: numeric argument required
2012-08-08 09:08:22 current leader's broker id : 255
2012-08-08 09:08:22 stopping server: 255
2012-08-08 09:08:22 sleeping for 10s

52. remove unused imports

53. ProducerPool.updateProducer: use a HashSet instead of ListBuffer for storing newBrokers.

54. The following unit test seems to fail occasionally for me. Is that related to this patch?
[error] Test Failed: testMultiProduceResend(kafka.integration.LazyInitProducerTest)
kafka.common.KafkaException: fetching broker partition metadata for topics [List(test1)] from
broker [ArrayBuffer(id:0,creatorId:192.168.1.111-1344443455425,host:192.168.1.111,port:61693)]
failed
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:109)
	at kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:49)
	at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:140)
	at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:101)
	at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:100)
	at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
	at scala.collection.immutable.List.foreach(List.scala:45)
	at kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:100)
	at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:65)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:48)
	at kafka.producer.Producer.send(Producer.scala:65)
	at kafka.integration.LazyInitProducerTest.testMultiProduceResend(LazyInitProducerTest.scala:163)

55. Could you change the comment for broker.list in ProducerConfig to the following?

  /** This is for bootstrapping and the producer will only use it for getting metadata (topics,
partitions and replicas).
   *  The socket connections for sending the actual data will be established based on the
broker information
   *  returned in the metadata. The format is host1:por1,host2:port2, and the list can be
a subset of brokers or
   *  a VIP pointing to a subset of brokers.
   */

                
> remove ZK dependency on producer
> --------------------------------
>
>                 Key: KAFKA-369
>                 URL: https://issues.apache.org/jira/browse/KAFKA-369
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: core
>    Affects Versions: 0.8
>            Reporter: Jun Rao
>            Assignee: Yang Ye
>         Attachments: kafka_369_v1.diff, kafka_369_v2.diff, kafka_369_v3.diff, kafka_369_v4.diff,
kafka_369_v5.diff
>
>   Original Estimate: 252h
>  Remaining Estimate: 252h
>
> Currently, the only place that ZK is actually used is in BrokerPartitionInfo. We use
ZK to get a list of brokers for making TopicMetadataRequest requests. Instead, we can provide
a list of brokers in the producer config directly. That way, the producer client is no longer
dependant on ZK.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message