kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bruno Cadonna (Jira)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-10686) Pluggable standby tasks assignor for Kafka Streams
Date Thu, 05 Nov 2020 13:56:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226725#comment-17226725

Bruno Cadonna commented on KAFKA-10686:

Hi [~lkokhreidze], I guess this ticket makes sense. Looking forward to your KIP!

You might want to look into KIP-441 to see how a Kafka Streams client passes its task lags
to the assignor. I can imagine that you want to use a similar mechanism to pass information
about a Kafka Streams client to the assignor.

> Pluggable standby tasks assignor for Kafka Streams
> --------------------------------------------------
>                 Key: KAFKA-10686
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10686
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Levani Kokhreidze
>            Priority: Major
> In production, Kafka Streams instances often run across different clusters and availability
zones. In order to guarantee high availability of the Kafka Streams deployments, users would
need more granular control over which on instances standby tasks can be created. 
> Idea of this ticket is to expose interface for Kafka Streams which can be implemented
by users to control where standby tasks can be created.
> Kafka Streams can have RackAware assignment as a default implementation that will take
into account `rack.id` of the application and make sure that standby tasks are created on
different racks. 
> Point of this ticket though is to give more flexibility to users on standby task creation,
in cases where just rack awareness is not enough. 

This message was sent by Atlassian Jira

View raw message