samza-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharath Kumarasubramanian (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SAMZA-2531) Async Commit is Blocking for Window, Timer & End of Stream
Date Tue, 26 May 2020 20:57:00 GMT

     [ https://issues.apache.org/jira/browse/SAMZA-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Bharath Kumarasubramanian updated SAMZA-2531:
---------------------------------------------
    Description: 
We introduced non-blocking commit feature as part of  SAMZA-1140. Currently the commit is
only non-blocking and allows process to be run concurrently.  

As part of this task, we will need to evaluate what it takes to make it run concurrently along
with window, timer & end of stream and if it is necessary in the first place. If not,
rename the feature to explicitly callout process exclusive commit instead of async commit. 

Some context:
 * Transactional state currently doesn't work well w/ async commit turned on. i.e. Even though
async commit, uses low watermark approach to find the offset to safely commit, it still doesn't
guarantee the state snapshots happen during the commit correspond to the offset that is committed
to checkpoint topic.
 * Window, timer could potentially mutate the state in the same manner as process; So it might
be okay to phase out (if we decide to support) the changes to introduce concurrency within
window & timer first and follow it up with synchronizing the state w/ the offset that
is committed. 

  was:
We introduced non-blocking commit feature as part of  SAMZA-1140. Currently the commit is
only non-blocking and allows process to be run concurrently.  

As part of this task, we will need to evaluate what it takes to make it run concurrently along
with window, timer & end of stream and if it is necessary in the first place. If not,
rename the feature to explicitly callout process exclusive commit instead of async commit. 


> Async Commit is Blocking for Window, Timer & End of Stream
> ----------------------------------------------------------
>
>                 Key: SAMZA-2531
>                 URL: https://issues.apache.org/jira/browse/SAMZA-2531
>             Project: Samza
>          Issue Type: Task
>            Reporter: Bharath Kumarasubramanian
>            Priority: Major
>
> We introduced non-blocking commit feature as part of  SAMZA-1140. Currently the commit
is only non-blocking and allows process to be run concurrently.  
> As part of this task, we will need to evaluate what it takes to make it run concurrently
along with window, timer & end of stream and if it is necessary in the first place. If
not, rename the feature to explicitly callout process exclusive commit instead of async commit. 
> Some context:
>  * Transactional state currently doesn't work well w/ async commit turned on. i.e. Even
though async commit, uses low watermark approach to find the offset to safely commit, it still
doesn't guarantee the state snapshots happen during the commit correspond to the offset that
is committed to checkpoint topic.
>  * Window, timer could potentially mutate the state in the same manner as process; So
it might be okay to phase out (if we decide to support) the changes to introduce concurrency
within window & timer first and follow it up with synchronizing the state w/ the offset
that is committed. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message