kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Roesler <j...@confluent.io>
Subject Re: [VOTE] KIP-328: Ability to suppress updates for KTables
Date Tue, 07 Aug 2018 17:07:45 GMT
Thanks everyone, KIP-328 has passed with 3 binding votes (Guozhang, Damian,
and Matthias) and 3 non-binding (Ted, Bill, and me).

Thanks for your time,
-John

On Mon, Aug 6, 2018 at 6:35 PM Matthias J. Sax <matthias@confluent.io>
wrote:

> +1 (binding)
>
> Thanks for the KIP.
>
>
> -Matthias
>
> On 8/3/18 12:52 AM, Damian Guy wrote:
> > Thanks John! +1
> >
> > On Mon, 30 Jul 2018 at 23:58 Guozhang Wang <wangguoz@gmail.com> wrote:
> >
> >> Yes, the addendum lgtm as well. Thanks!
> >>
> >> On Mon, Jul 30, 2018 at 3:34 PM, John Roesler <john@confluent.io>
> wrote:
> >>
> >>> Another thing that came up after I started working on an implementation
> >> is
> >>> that in addition to deprecating "retention" from the Windows interface,
> >> we
> >>> also need to deprecate "segmentInterval", for the same reasons. I
> simply
> >>> overlooked it previously. I've updated the KIP accordingly.
> >>>
> >>> Hopefully, this doesn't change anyone's vote.
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> On Mon, Jul 30, 2018 at 5:31 PM John Roesler <john@confluent.io>
> wrote:
> >>>
> >>>> Thanks Guozhang,
> >>>>
> >>>> Thanks for that catch. to clarify, currently, events are "late" only
> >> when
> >>>> they are older than the retention period. Currently, we detect this
in
> >>> the
> >>>> processor and record it as a "skipped-record". We then do not attempt
> >> to
> >>>> store the event in the window store. If a user provided a
> >> pre-configured
> >>>> window store with a retention period smaller than the one they specify
> >>> via
> >>>> Windows#until, the segmented store will drop the update with no metric
> >>> and
> >>>> record a debug-level log.
> >>>>
> >>>> With KIP-328, with the introduction of "grace period" and moving
> >>> retention
> >>>> fully into the state store, we need to have metrics for both "late
> >>> events"
> >>>> (new records older than the grace period) and "expired window events"
> >>> (new
> >>>> records for windows that are no longer retained in the state store).
I
> >>>> already proposed metrics for the late events, and I've just updated
> the
> >>> KIP
> >>>> with metrics for the expired window events. I also updated the KIP to
> >>> make
> >>>> it clear that neither late nor expired events will count as
> >>>> "skipped-records" any more.
> >>>>
> >>>> -John
> >>>>
> >>>> On Mon, Jul 30, 2018 at 4:22 PM Guozhang Wang <wangguoz@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Hi John,
> >>>>>
> >>>>> Thanks for the updated KIP, +1 from me, and one minor suggestion:
> >>>>>
> >>>>> Following your suggestion of the differentiation of `skipped-records`
> >>> v.s.
> >>>>> `late-event-drop`, we should probably consider moving the scenarios
> >>> where
> >>>>> records got ignored due the window not being available any more
in
> >>>>> windowed
> >>>>> aggregation operators from the `skipped-records` metrics recording
to
> >>> the
> >>>>> `late-event-drop` metrics recording.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 30, 2018 at 1:36 PM, Bill Bejeck <bbejeck@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> Thanks for the KIP!
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> -Bill
> >>>>>>
> >>>>>> On Mon, Jul 30, 2018 at 3:42 PM Ted Yu <yuzhihong@gmail.com>
wrote:
> >>>>>>
> >>>>>>> +1
> >>>>>>>
> >>>>>>> On Mon, Jul 30, 2018 at 11:46 AM John Roesler <john@confluent.io>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Hello devs,
> >>>>>>>>
> >>>>>>>> The discussion of KIP-328 has gone some time with no
new
> >> comments,
> >>>>> so I
> >>>>>>> am
> >>>>>>>> calling for a vote!
> >>>>>>>>
> >>>>>>>> Here's the KIP: https://cwiki.apache.org/confluence/x/sQU0BQ
> >>>>>>>>
> >>>>>>>> The basic idea is to provide:
> >>>>>>>> * more usable control over update rate (vs the current
state
> >> store
> >>>>>>> caches)
> >>>>>>>> * the final-result-for-windowed-computations feature
which
> >>> several
> >>>>>> people
> >>>>>>>> have requested
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> -John
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message