kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6047) Allow retries configuration for InternalTopicManager
Date Mon, 16 Oct 2017 23:33:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206775#comment-16206775
] 

Guozhang Wang commented on KAFKA-6047:
--------------------------------------

Thanks for reporting this [~dvsekhvalnov]. I agree that we should make Streams API be more
resilient with such broker unavailability issues. Regarding whether we should add a separate
configuration to cover this case, maybe not need to. [~mjsax] is already looking into this
issue, and I'll leave it to him to present a proposal.

> Allow retries configuration for InternalTopicManager
> ----------------------------------------------------
>
>                 Key: KAFKA-6047
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6047
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 0.11.0.0
>            Reporter: Dmitry Vsekhvalnov
>            Priority: Minor
>
> There is hardcoded number of retries when kafka-streams attempts to create internal topics.
> *InternalTopicManager.MAX_TOPIC_READY_TRY*
> Which is not resilient in some scenarios. Consider setup where replication factor for
internal streams topics == number of brokers. (RF=3 and x3 kafka brokers). When any of brokers
dies kafka-streams can shutdown before broker is resurrected with approximate log:
> {code}
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] [Could not
create internal topics: Found only 2 brokers,  but replication factor is 3. Decrease replication
factor for internal topics via StreamsConfig parameter "replication.factor" or add more brokers
to your cluster. Retry #2]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] [Could not
create internal topics: Found only 2 brokers,  but replication factor is 3. Decrease replication
factor for internal topics via StreamsConfig parameter "replication.factor" or add more brokers
to your cluster. Retry #3]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager] [Could not
create internal topics: Found only 2 brokers,  but replication factor is 3. Decrease replication
factor for internal topics via StreamsConfig parameter "replication.factor" or add more brokers
to your cluster. Retry #4]
> [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread] [stream-thread [Shutting
down]
> {code}
> Would be nice if kafka-streams provides configuration for InternalTopicManager retries
and ideally for retry backoff strategy. To have possibility for resilience tuning. Possibly
can re-use corresponding producer configuration. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message