Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
To: users@kafka.apache.org, "dev@kafka.apache.org" <dev@kafka.apache.org>
References: <21df6e5c-3aed-ac03-339a-0c7526c4392a@confluent.io>
 <CAHoiPjymsUhTjHWwogLc3Vo=fQAmD--Tu9vHTfQ6ak7j8MYF9g@mail.gmail.com>
 <CAJikTEVez6Okzinu8yhDcS_zWTD4zj2wM2ZZwXvVUEP0-EUjQg@mail.gmail.com>
 <87c6103a-08b9-b895-d8bd-5fbd5885b550@confluent.io>
 <CAHoiPjzkwZtvfrSJ=nSpcb8X5vv_HOw6CQ+3rwg-g7C_Kzz4Uw@mail.gmail.com>
 <CAHwHRrUQbKj0Y3dLyP0rz3Yro59CyMAa83Rtv01xJvTuuCAYVA@mail.gmail.com>
 <CAHoiPjxbPkO2yjv1-KOPQS7AHy=WJrFLsNq=rfgV-GODffBesw@mail.gmail.com>
 <CAHwHRrUSy2ymQP27x5jU=v77u-cozSMZ2=8ZmfwkW0cvxdAxRw@mail.gmail.com>
 <2798abbf-02da-6f1c-f4d9-3ff4e238fab0@confluent.io>
From: "Matthias J. Sax" <matthias@confluent.io>
Organization: Confluent Inc
Message-ID: <2ba66790-e3c9-a3d1-a801-b3835a3a3559@confluent.io>
Date: Wed, 15 Feb 2017 16:04:54 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0)
 Gecko/20100101 Thunderbird/45.7.1
MIME-Version: 1.0
In-Reply-To: <2798abbf-02da-6f1c-f4d9-3ff4e238fab0@confluent.io>
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature";
 boundary="VAgHLaTue8eT4AdgEKgkBLMt2EwOd6Pq1"
archived-at: Thu, 16 Feb 2017 00:05:24 -0000

--VAgHLaTue8eT4AdgEKgkBLMt2EwOd6Pq1
Content-Type: multipart/mixed; boundary="213eSAc1ouI6GdJmoQnwRkaIWJXSHNltX";
 protected-headers="v1"
From: "Matthias J. Sax" <matthias@confluent.io>
To: users@kafka.apache.org, "dev@kafka.apache.org" <dev@kafka.apache.org>
Message-ID: <2ba66790-e3c9-a3d1-a801-b3835a3a3559@confluent.io>
Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
References: <21df6e5c-3aed-ac03-339a-0c7526c4392a@confluent.io>
 <CAHoiPjymsUhTjHWwogLc3Vo=fQAmD--Tu9vHTfQ6ak7j8MYF9g@mail.gmail.com>
 <CAJikTEVez6Okzinu8yhDcS_zWTD4zj2wM2ZZwXvVUEP0-EUjQg@mail.gmail.com>
 <87c6103a-08b9-b895-d8bd-5fbd5885b550@confluent.io>
 <CAHoiPjzkwZtvfrSJ=nSpcb8X5vv_HOw6CQ+3rwg-g7C_Kzz4Uw@mail.gmail.com>
 <CAHwHRrUQbKj0Y3dLyP0rz3Yro59CyMAa83Rtv01xJvTuuCAYVA@mail.gmail.com>
 <CAHoiPjxbPkO2yjv1-KOPQS7AHy=WJrFLsNq=rfgV-GODffBesw@mail.gmail.com>
 <CAHwHRrUSy2ymQP27x5jU=v77u-cozSMZ2=8ZmfwkW0cvxdAxRw@mail.gmail.com>
 <2798abbf-02da-6f1c-f4d9-3ff4e238fab0@confluent.io>
In-Reply-To: <2798abbf-02da-6f1c-f4d9-3ff4e238fab0@confluent.io>

--213eSAc1ouI6GdJmoQnwRkaIWJXSHNltX
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Hi,

according to the feedback, I updated the KIP, and limited its scope to
some extend:

- Instead of changing the creation of KafkaStreams instances, we keep
the current pattern (we might do a follow up KIP on this though).

- We also added a new method #describe() that returns a
TopologyDescription that can be used for any "pretty print"
functionality or similar.

- We also removed method #topologyBuilder() from KStreamBuilder because
we think #transform() should provide all functionality you need to
mix-an-match Processor API and DSL. If there is any further concern
about this, please let us know.


-Matthias


On 2/14/17 9:59 AM, Matthias J. Sax wrote:
> You can already output any number of record within .transform() using
> the provided Context object from init()...
>=20
>=20
> -Matthias
>=20
> On 2/14/17 9:16 AM, Guozhang Wang wrote:
>>> and you can't output multiple records or branching logic from a
>> transform();
>>
>> For output multiple records in transform, we are currently working on
>> https://issues.apache.org/jira/browse/KAFKA-4217, I think that should =
cover
>> this use case.
>>
>> For branching the output in transform, I agree this is not perfect but=
 I
>> think users can follow some patterns like "stream.transform().branch()=
",
>> would that work for you?
>>
>>
>> Guozhang
>>
>>
>> On Tue, Feb 14, 2017 at 8:29 AM, Mathieu Fenniak <
>> mathieu.fenniak@replicon.com> wrote:
>>
>>> On Tue, Feb 14, 2017 at 1:14 AM, Guozhang Wang <wangguoz@gmail.com> w=
rote:
>>>
>>>> Some thoughts on the mixture usage of DSL / PAPI:
>>>>
>>>> There were some suggestions on mixing the usage of DSL and PAPI:
>>>> https://issues.apache.org/jira/browse/KAFKA-3455, and after thinking=
 it
>>> a
>>>> bit more carefully, I'd rather not recommend users following this
>>> pattern,
>>>> since in DSL this can always be achieved in process() / transform().=

>>> Hence
>>>> I think it is okay to prevent such patterns in the new APIs. And for=
 the
>>>> same reasons, I think we can remove KStreamBuilder#newName() from th=
e
>>>> public APIs.
>>>>
>>>
>>> I'm not sure that things can always be achieved by process() /
>>> transform()... there are some limitations to these APIs.  You can't o=
utput
>>> from a process(), and you can't output multiple records or branching =
logic
>>> from a transform(); these are things that can be done in the PAPI qui=
te
>>> easily.
>>>
>>> I definitely understand a preference for using process()/transform() =
where
>>> possible, but, they don't seem to replace the PAPI.
>>>
>>> I would love to be operating in a world that was entirely DSL.  But t=
he DSL
>>> is limited, and it isn't extensible (... by any stable API).  I don't=
 mind
>>> reaching into internals today and making my own life difficult to ext=
end
>>> it, and I'd continue to find a way to do that if you made the APIs di=
stinct
>>> and split, but I'm just expressing my preference that you not do that=
=2E :-)
>>>
>>> And about printing the topology for debuggability: I agrees this is a=

>>>> potential drawback, and I'd suggest maintain some functionality to b=
uild
>>> a
>>>> "dry topology" as Mathieu suggested; the difficulty is that, interna=
lly
>>> we
>>>> need a different "copy" of the topology for each thread so that they=
 will
>>>> not share any states, so we cannot directly pass in the topology int=
o
>>>> KafkaStreams instead of the topology builder. So how about adding a
>>>> `ToplogyBuilder#toString` function which calls `build()` internally =
then
>>>> prints the built dry topology?
>>>>
>>>
>>> Well, this sounds better than KafkaStreams#toString() in that it does=
n't
>>> require a running processor.  But I'd really love to have a simple ob=
ject
>>> model for the topology, not a string output, so that I can output my =
own
>>> debug format.  I currently have that in the form of
>>> TopologyBuilder#nodeGroups() & TopologyBuilder#build(Integer).
>>>
>>> Mathieu
>>>
>>
>>
>>
>=20


--213eSAc1ouI6GdJmoQnwRkaIWJXSHNltX--

--VAgHLaTue8eT4AdgEKgkBLMt2EwOd6Pq1
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJYpOynAAoJECnhiMLycopPjl4P/2qgbwN003PP6zdbjWdadJrO
tzwIVj7H5cjQQwgzEeqlbHGTXTL80Z1Slt1fIqR+VqssYGCPOWiKp3lix7wBAcud
JmnvhKZl3iBLqWm3qFJaz9nA6ckszY5ciQ/TGeJpjejfLs9rJfny/6LYRioHE079
o7upzZKUECoU4VSf6lDuNoHQD79VCD3oKZP1vs/1GsZLYrOvUDuIZtW80WIAiPfw
1VAK2pfIkmC+9arNyDMSJyc//n9+2lFPZWlAkrclQcPBcA/YOcdfl7tZ3O2IdDaG
S09ggpnuNgroavs62vs/QTuBNoYiZF98hLLyETBG1cDc3v/BORY5CkIuK8NK+rJ5
kPVYbE9mEKpAi5VXkvfVEs9rmImqlRLxK8/WXEqcyqB/enVyHD2yOTBwqybDzUAH
NJQD8OXpLmt1BFCVg6iukOGD72XFR8GqH6lqFYHHTC6A/WbzuaQJ6zDrbbsuvtfB
BWxqgDCFqoNbOVcdx3S/+etQ2sKgpqXdDC6CdUmLDJA6yAhxK4Z98qA7ZkBkH7NK
lA8neLXxetwWpSiu4Nld0uxudqztf0r4RHAP9QGbuK83gaaW0nLm2UxNaDd0tLCn
lc4zI+Ai+56nAbfVCZqPRGlDshNioVObc1v5xdALvTQ20z8Jgx0qwCk6b3qtiy3q
y/sJdWVPxh2QHGWnjLZ/
=OYXp
-----END PGP SIGNATURE-----

--VAgHLaTue8eT4AdgEKgkBLMt2EwOd6Pq1--