kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4677) Avoid unnecessary task movement across threads during rebalance
Date Wed, 01 Mar 2017 21:01:45 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891031#comment-15891031

Matthias J. Sax commented on KAFKA-4677:

Streams used it's own partition assigner implementation and does not use the Consumer's default
partition assigner, because streams has additional requirements/restrictions with regard to
group management. Thus, any change from KIP-54 will not affect Streams as it does not use

> Avoid unnecessary task movement across threads during rebalance
> ---------------------------------------------------------------
>                 Key: KAFKA-4677
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4677
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions:
>            Reporter: Damian Guy
>            Assignee: Damian Guy
>             Fix For:
> StreamPartitionAssigner tries to follow a sticky assignment policy to avoid expensive
task migration. Currently, it does this in a best-effort approach.
> We could observe a case, for which tasks did migrate for no good reason, thus we assume
that the current implementation could be improved to be more sticky.
> The concrete scenario is as follows:
> assume we have topology with 3 tasks, A, B, C
> assume we have 3 threads, each executing one task: 1-A, 2-B, 3-C
> for some reason, thread 1 goes down and a rebalance gets triggered
> thread 2 and 3 get their partitions revoked
> sometimes (not sure what the exact condition for this is), the new assignment flips the
assignment for task B and C (task A is newly assigned to either thread 2 or 3)
> > possible new assignment 2(A,C) and 3-B
> There is no obvious reason (like load-balancing) why the task assignment for B and C
does change to the other thread resulting in unnecessary task migration.

This message was sent by Atlassian JIRA

View raw message