Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
References: <CAERNSNce19QsFhJVvyxyDJC6XAkY0yRBdp07D6xnzsHUB+Fg_g@mail.gmail.com>
 <CAERNSNdcP1E_EAMF1=Z1hG63o=qR__hT6kh4tmUaJtuDo9oAEw@mail.gmail.com>
 <CA+rPSbYXtMVfV+H-oBKfanzmh7HYUcwwgzd-U15eBO_szJt27g@mail.gmail.com> <CAHwHRrUdYNgp55Yw2wZmi=msbVfHyum_9yfgCS651vQGuZfoJQ@mail.gmail.com>
In-Reply-To: <CAHwHRrUdYNgp55Yw2wZmi=msbVfHyum_9yfgCS651vQGuZfoJQ@mail.gmail.com>
From: =?UTF-8?B?WGF2aWVyIEzDqWF1dMOp?= <xavier@confluent.io>
Date: Wed, 24 May 2017 05:44:45 +0000
Message-ID: <CA+rPSbaYPv53k8cq8BWJNwkYCf_fbBHe3h2i+reSr-xUTGVRSw@mail.gmail.com>
Subject: Re: [Vote] KIP-150 - Kafka-Streams Cogroup
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Content-Type: multipart/alternative; boundary="001a114e739a9dbd0b05503e9a3c"
archived-at: Wed, 24 May 2017 05:45:03 -0000

--001a114e739a9dbd0b05503e9a3c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I don't think we should wait for entries from each stream, since that might
limit the usefulness of the cogroup operator. There are instances where it
can be useful to compute something based on data from one or more stream,
without having to wait for all the streams to produce something for the
group. In the example I gave in the discussion, it is possible to compute
impression/auction statistics without having to wait for click data, which
can typically arrive several minutes late.

We could have a separate discussion around adding inner / outer modifiers
to each of the streams to decide which fields are optional / required
before sending updates if we think that might be useful.


On Tue, May 23, 2017 at 6:28 PM Guozhang Wang <wangguoz@gmail.com> wrote:

> The proposal LGTM, +1
>
> One question I have is about when to send the record to the resulted KTab=
le
> changelog. For example in your code snippet in the wiki page, before you
> see the end result of
>
> 1L, Customer[
>
>                       cart:{Item[no:01], Item[no:03], Item[no:04]},
>                       purchases:{Item[no:07], Item[no:08]},
>                       wishList:{Item[no:11]}
>       ]
>
>
> You will firs see
>
> 1L, Customer[
>
>                       cart:{Item[no:01]},
>                       purchases:{},
>                       wishList:{}
>       ]
>
> 1L, Customer[
>
>                       cart:{Item[no:01]},
>                       purchases:{Item[no:07],Item[no:08]},
>
>                       wishList:{}
>       ]
>
> 1L, Customer[
>
>                       cart:{Item[no:01]},
>                       purchases:{Item[no:07],Item[no:08]},
>
>                       wishList:{}
>       ]
>
> ...
>
>
> I'm wondering if it makes more sense to only start sending the update if
> the corresponding agg-key has seen at least one input from each of the
> input stream? Maybe it is out of the scope of this KIP and we can make it=
 a
> more general discussion in a separate one.
>
>
> Guozhang
>
>
> On Fri, May 19, 2017 at 8:37 AM, Xavier L=C3=A9aut=C3=A9 <xavier@confluen=
t.io>
> wrote:
>
> > Hi Kyle, I left a few more comments in the discussion thread, if you
> > wouldn't mind taking a look
> >
> > On Fri, May 19, 2017 at 5:31 AM Kyle Winkelman <winkelman.kyle@gmail.co=
m
> >
> > wrote:
> >
> > > Hello all,
> > >
> > > I would like to start the vote on KIP-150.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+
> > Kafka-Streams+Cogroup
> > >
> > > Thanks,
> > > Kyle
> > >
> >
>
>
>
> --
> -- Guozhang
>

--001a114e739a9dbd0b05503e9a3c--