kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6047) Allow retries configuration for InternalTopicManager
Date Sun, 15 Oct 2017 13:20:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205108#comment-16205108
] 

Ted Yu commented on KAFKA-6047:
-------------------------------

Does this require a KIP ?

> Allow retries configuration for InternalTopicManager
> ----------------------------------------------------
>
>                 Key: KAFKA-6047
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6047
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 0.11.0.0
>            Reporter: Dmitry Vsekhvalnov
>            Priority: Minor
>
> There is hardcoded number of retries when kafka-streams attempts to create internal topics.
> *InternalTopicManager.MAX_TOPIC_READY_TRY*
> Which is not resilient in some scenarios. Consider setup where replication factor for
internal streams topics == number of brokers. (RF=3 and x3 kafka brokers). When any of brokers
dies kafka-streams can shutdown before broker is resurrected with approximate log:
> {code}
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] [Could not
create internal topics: Found only 2 brokers,  but replication factor is 3. Decrease replication
factor for internal topics via StreamsConfig parameter "replication.factor" or add more brokers
to your cluster. Retry #2]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] [Could not
create internal topics: Found only 2 brokers,  but replication factor is 3. Decrease replication
factor for internal topics via StreamsConfig parameter "replication.factor" or add more brokers
to your cluster. Retry #3]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] [Could not
create internal topics: Found only 2 brokers,  but replication factor is 3. Decrease replication
factor for internal topics via StreamsConfig parameter "replication.factor" or add more brokers
to your cluster. Retry #4]
> [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread] [stream-thread [Shutting
down]
> {code}
> Would be nice if kafka-streams provides configuration for InternalTopicManager retries
and ideally for retry backoff strategy. To have possibility for resilience tuning. Possibly
can re-use corresponding producer configuration. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message