kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4969) State-store workload-aware StreamsPartitionAssignor
Date Thu, 15 Feb 2018 21:49:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366299#comment-16366299

Matthias J. Sax commented on KAFKA-4969:

Should we really close this JIRA? I think we could do even better and the description covers
more than addressed so far. (As alternative, we can also create a new ticket, and keep this
one with a limited scope). WDYT?

> State-store workload-aware StreamsPartitionAssignor
> ---------------------------------------------------
>                 Key: KAFKA-4969
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4969
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: streams
>            Reporter: Matthias J. Sax
>            Assignee: Bill Bejeck
>            Priority: Major
>             Fix For: 1.1.0, 1.0.1
> Currently, {{StreamPartitionsAssigner}} does not distinguish different "types" of tasks.
For example, task can be stateless of have one or multiple stores.
> This can lead to an suboptimal task placement: assume there are 2 stateless and 2 stateful
tasks and the app is running with 2 instances. To share the "store load" it would be good
to place one stateless and one stateful task per instance. Right now, there is no guarantee
about this, and it can happen, that one instance processed both stateless tasks while the
other processes both stateful tasks.
> We should improve {{StreamPartitionAssignor}} and introduce "task types" including a
cost model for task placement. We should consider the following parameters:
>  - number of stores
>  - number of sources/sinks
>  - number of processors
>  - regular task vs standby task
> This improvement should be backed by a design document in the project wiki (no KIP required
though) as it's a fairly complex change.

This message was sent by Atlassian JIRA

View raw message