kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Surana (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-6555) Making state store queryable during restoration
Date Thu, 15 Feb 2018 23:43:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361515#comment-16361515
] 

Ashish Surana edited comment on KAFKA-6555 at 2/15/18 11:42 PM:
----------------------------------------------------------------

Hi Matthias,

  Thanks for pointing to KAFKA-6144. As I go through it, it looks very similar ticket but
with minor difference:
 * I am suggesting to allow stale reads only from PARTITION_ASSIGNED (not from PARTITION_REVOKED)
primarily as it's going to be the one in RUNNING state, and this is the minimum we need to
do keep serving request for this partition. We still have only one instance doing read/write,
and will have pure standby replicas. This approach is good if we continue with current design
of one active write & read instance.

I have made the changes, and can share it in few days.


was (Author: asurana):
Hi Matthias,

  Thanks for pointing to KAFKA-6144. As I go through it, it looks very similar ticket but
with minor difference:
 * I am suggesting to allow stale reads only from PARTITION_ASSIGNED (not from PARTITION_REVOKED)
primarily as it's going to be the one in RUNNING state, and this is the minimum we need to
do keep serving request for this partition. We still have one instance doing write or read
and still want to have pure standby replicas. This approach is good if we continue with current
design of one active write/read instance.

I have made the changes, and can share it in few days.

> Making state store queryable during restoration
> -----------------------------------------------
>
>                 Key: KAFKA-6555
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6555
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Ashish Surana
>            Priority: Major
>
> State store in Kafka streams are currently only queryable when StreamTask is in RUNNING
state. The idea is to make it queryable even in the RESTORATION (PARTITION_ASSIGNED) state
as the time spend on restoration can be huge and making the data inaccessible during this
time could be downtime not suitable for many applications.
> When the active partition goes down then one of the following occurs:
>  # One of the standby replica partition gets promoted to active: Replica task has to
restore the remaining state from the changelog topic before it can become RUNNING. The time
taken for this depends on how much the replica is lagging behind. During this restoration
time the state store for that partition is currently not queryable resulting in the partition
downtime. We can make the state store partition queryable for the data already present in
the state store.
>  # When there is no replica or standby task, then active task will be started in one
of the existing node. That node has to build the entire state from the changelog topic which
can take lot of time depending on how big is the changelog topic, and keeping state store
not queryable during this time is the downtime for the parition.
> It's very important improvement as it could simply improve the availability of microservices
developed using kafka streams.
> I am working on a patch for this change. Any feedback or comments are welcome.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message