kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter K. (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-1182) Topic not created if number of live brokers less than # replicas
Date Mon, 27 Mar 2017 05:09:41 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942645#comment-15942645
] 

Peter K. edited comment on KAFKA-1182 at 3/27/17 5:09 AM:
----------------------------------------------------------

Guarantees (last version of docs) are not completely kept, while we have this issue.
"At a high-level Kafka gives the following guarantees:
...
For a topic with replication factor N, we will tolerate up to N-1 server failures without
losing any messages committed to the log. "

New topics simply not created, producer fails.
For scenarios where dynamic topics are the must  - this is a serious problem.

Is there any progress on that issue? I see the same in Kafka 0.9








was (Author: beep02):
Guarantees (last version of docs) are not completely kept, while we have this issue.
"At a high-level Kafka gives the following guarantees:
...
For a topic with replication factor N, we will tolerate up to N-1 server failures without
losing any messages committed to the log. "

New topics simply not created, producer fails.
For scenarios where dynamic topics are the must  - this is a serious problem.

Is there any progress on that issue? I see the same in Kafka 0.9
Did not check yet Kafka 1.0...






> Topic not created if number of live brokers less than # replicas
> ----------------------------------------------------------------
>
>                 Key: KAFKA-1182
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1182
>             Project: Kafka
>          Issue Type: Improvement
>          Components: producer 
>    Affects Versions: 0.8.0
>         Environment: Centos 6.3
>            Reporter: Hanish Bansal
>            Assignee: Jun Rao
>
> We are having kafka cluster of 2 nodes. (Using Kafka 0.8.0 version)
> Replication Factor: 2
> Number of partitions: 2
> Actual Behaviour:
> Out of two nodes, if any one node goes down then topic is not created in kafka. 
> Steps to Reproduce:
> 1. Create a 2 node kafka cluster with replication factor 2
> 2. Start the Kafka cluster
> 3. Kill any one node
> 4.  Start the producer to write on a new topic
> 5. Observe the exception stated below:
> 2013-12-12 19:37:19 0 [WARN ] ClientUtils$ - Fetching topic metadata with
> correlation id 3 for topics [Set(test-topic)] from broker
> [id:0,host:122.98.12.11,port:9092] failed
> java.net.ConnectException: Connection refused
>     at sun.nio.ch.Net.connect(Native Method)
>     at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
>     at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>     at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
>     at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
>     at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
>     at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
>     at
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
>     at
> kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:49)
>     at
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:186)
>     at
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:150)
>     at
> kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
>     at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
>     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
>     at
> kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:149)
>     at
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:95)
>     at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
>     at kafka.producer.Producer.send(Producer.scala:76)
>     at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> Expected Behaviour: 
> In case of live brokers less than # replicas:
> There should be topic created so at least live brokers can receive the data.
> They can replicate data to other broker once any down broker comes up.
> Because now in case of live brokers less than # replicas, there is complete
> loss of data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message