kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4667) Connect should create internal topics
Date Wed, 03 May 2017 04:46:04 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994268#comment-15994268

Ewen Cheslack-Postava commented on KAFKA-4667:

[~rhauch] Discussed some of these issues offline, but there are some other refinements here

* based on KIP-115, default replication factor of 3 but override it in our demo configs to
make running on a single local node easy is probably the least likely to cause unexpected
* partitions = 1 is only really true for config topic where ordering between differing keys
is required. For status and offsets topics you can go higher, and Confluent recommends doing
so: http://docs.confluent.io/current/connect/userguide.html#distributed-mode
* Not sure if we should fiddle with unclean leader election defaults. But maybe it is ok to
set this to false and tell users if they want the unsafe version they need to set up the topic
themselves before starting Connect? Not sure what the right tradeoff of respecting users'
existing settings vs encouraging better behavior is. This might even differ between the three
topics -- status topics are way less critical than config topic.

> Connect should create internal topics
> -------------------------------------
>                 Key: KAFKA-4667
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4667
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>            Reporter: Emanuele Cesena
>            Priority: Critical
>             Fix For:
> I'm reporting this as an issue but in fact it requires more investigation (which unfortunately
I'm not able to perform at this time).
> Repro steps:
> - configure Kafka for consistency, for example:
> default.replication.factor=3
> min.insync.replicas=2
> unclean.leader.election.enable=false
> - run Connect for the first time, which should create its internal topics
> I believe these topics are created with the broker's default, in particular:
> min.insync.replicas=2
> unclean.leader.election.enable=false
> but connect doesn't produce with acks=all, which in turn may cause the cluster to go
in a bad state (see, e.g., https://issues.apache.org/jira/browse/KAFKA-4666).
> Solution would be to force availability mode, i.e. force:
> unclean.leader.election.enable=true
> when creating the connect topics, or viceversa detect availability vs consistency mode
and turn acks=all if needed.
> I assume the same happens with other kafka-based services such as streams.

This message was sent by Atlassian JIRA

View raw message