samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Exactly once processing
Date Fri, 15 Apr 2016 16:37:41 GMT
Hi Sab,

For stateful processing where you have persistent state stores, you need to
maintain the checkpoint which includes the committed offsets as well as the
store flushed in sync, but right not these two operations are not done
atomically, and hence if you fail in between, you could still get
duplicates where you consume from the committed offsets while some of them
have already updated the stores.

Guozhang


On Thu, Apr 14, 2016 at 11:56 PM, Sasidharan, Sabarish <
Sabarish.Sasidharan@harman.com> wrote:

> Hi
>
> To achieve exactly once processing for my aggregates, wouldn’t it be
> enough if I maintain the latest offset processed for the aggregate and
> check against that offset when messages are replayed on recovery? Am I
> missing something here?
>
> Thanks
>
> Regards
> Sab




-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message