kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apurva Mehta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5666) Need feedback to user if consumption fails due to offsets.topic.replication.factor=3
Date Thu, 27 Jul 2017 18:19:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103644#comment-16103644

Apurva Mehta commented on KAFKA-5666:

A solution for just the console consumer could be to have some precondition checks before
entering the IO loop. 

In this case, we could check whether the offset topics exists using the admin client ([~yevabyzek]'s

If these preconditions are not met after a certain amount of time, the console consumer could
exit with a proper error about which precondition failed. 

This seems like a better route for solving the developer experience problem: check for preconditions
in the app itself because the environment for these applications is more likely to have problems
relative to a prod environment.

> Need feedback to user if consumption fails due to offsets.topic.replication.factor=3
> ------------------------------------------------------------------------------------
>                 Key: KAFKA-5666
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5666
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, consumer
>    Affects Versions:
>            Reporter: Yeva Byzek
>              Labels: newbie
> Introduced in 0.11: The offsets.topic.replication.factor broker config is now enforced
upon auto topic creation. Internal auto topic creation will fail with a GROUP_COORDINATOR_NOT_AVAILABLE
error until the cluster size meets this replication factor requirement.
> Issue: Default is setting offsets.topic.replication.factor=3, but in development and
docker environments where there is only 1 broker, the offsets topic will fail to be created
when a consumer tries to consume and no records will be returned.  As a result, the user experience
is bad.  The user may have no idea about this setting change and enforcement, and they just
see that `kafka-console-consumer` hangs with ZERO output. It is true that the broker log file
will provide a message (e.g. {{ERROR [KafkaApi-1] Number of alive brokers '1' does not meet
the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor').
This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)}})
but many users do not have access to the log files or know how to get them.
> Suggestion: give feedback to the user/app if offsets topic cannot be created.  For example,
after some timeout.
> Workaround:
> Set offsets.topic.replication.factor=3

This message was sent by Atlassian JIRA

View raw message