kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyang Chen <bche...@outlook.com>
Subject Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
Date Mon, 29 Apr 2019 20:32:54 GMT
Could we get more discussions on this thread?


From: Boyang Chen <bchen11@outlook.com>
Sent: Friday, April 26, 2019 10:51 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks for the explanation Matthias! Will enhance the KIP motivation by your example.

From: Matthias J. Sax <matthias@confluent.io>
Sent: Friday, April 26, 2019 3:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks for the KIP!

I agree that the change makes sense, and not only for the static group
membership case.

For example, if a user `closes()` a `KafkaStreams` client and creates a
new one (for example to recover failed threads), while the JVM is still
running, it is more intuitive that the thread names are number from 1 to
X again, and not from X+1 to 2*x on restart.

Also, the original idea about making thread names unique across
application is non-intuitive itself. It might make sense if there are
two instances of the same application within one JVM -- however, this
seems to be a rather rare case. Also, the only pattern for this use case
seems to by dynamic scaling, and I believe we should actually void this
pattern by adding a `stopThread()` and `addThread()` method to
`KafkaStreams` directly.


On 4/25/19 11:13 PM, Boyang Chen wrote:
> Hey friends,
> I would like to start discussion for a very small KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> it is trying to avoid sharing thread-id increment between multiple stream instances configured
in one JVM. This is an important fix for static membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
to be effective for KStreams in edge case like changing `group.instance.id` throughout restarts
due to thread-id interleaving.
> I will open the vote thread in the main while, since this is a very small fix. Feel free
to continue the discussion on this thread, thank you!
> Boyang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message