kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Giridhar Addepalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6645) Host Affinity to facilitate faster restarts of kafka streams applications
Date Wed, 14 Mar 2018 06:35:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398148#comment-16398148

Giridhar Addepalli commented on KAFKA-6645:

Thank you for your reply [~mjsax]

Can you please provide us with code pointer for this.

Is there some duration of time for which leader of consumer group waits for all the consumers
to join the group ?

If Kafka Streams application is running on say 10 machines before, and we stopped the application
on all machines now.

Now, say we are in the process of bringing up the application on the machines. During this
it should not be the case that kafka streams thinks that other machines are down and try to
assign partitions within the machines that are currently up.

> Host Affinity to facilitate faster restarts of kafka streams applications
> -------------------------------------------------------------------------
>                 Key: KAFKA-6645
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6645
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: Giridhar Addepalli
>            Priority: Major
> Since Kafka Streams applications have lot of state in the stores in general, it would
be good to remember the assignment of partitions to machines. So that when whole application
is restarted for some reason, there is a way to use past assignment of partitions to machines
and there won't be need to build up whole state by reading off of changelog kafka topic. This
would result in faster start-up.
> Samza has support for Host Affinity ([https://samza.apache.org/learn/documentation/0.14/yarn/yarn-host-affinity.html])
> KIP-54 ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy)] ,
handles cases where some members of consumer group goes down / comes up, and KIP-54 ensures
there is minimal diff between assignments before and after rebalance. 
> But to handle whole restart use case, we need to remember past assignment somewhere,
and use it after restart.
> Please let us know if this is already solved problem / some cleaner way of achieving
this objective

This message was sent by Atlassian JIRA

View raw message