Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
MIME-Version: 1.0
In-Reply-To: <CAJikTEVFLTarqQwv+7LiqdwgG_v8zXyMU+NFxH10xhyCKM6h_g@mail.gmail.com>
References: <CADu8Qr=me+7uKiAstJ2Ot=MaJ1HTB95Un_i=9Lu95UObQagjkg@mail.gmail.com>
 <08a1d28d-a157-11a5-f9ca-91635066ffc7@confluent.io> <CAJikTEVFLTarqQwv+7LiqdwgG_v8zXyMU+NFxH10xhyCKM6h_g@mail.gmail.com>
From: Florian Hussonnois <fhussonnois@gmail.com>
Date: Tue, 18 Oct 2016 16:36:34 +0200
Message-ID: <CADu8QrmC3YT18im2-WFDKcFQHncou-7+5J7hueXMkA0Ota4OUA@mail.gmail.com>
Subject: Re: KStreams / add support for sink processor with dynamic topics
To: dev@kafka.apache.org
Content-Type: multipart/alternative; boundary=089e0103e2f696092b053f249e65
archived-at: Tue, 18 Oct 2016 14:36:42 -0000

--089e0103e2f696092b053f249e65
Content-Type: text/plain; charset=UTF-8

Thank you Matthias for your answers.

The mailing list that you linked shows a solution using the Processor API.

Actually, the set of subtypes is not known in advance this is why I need to
compute output topics from messages. So the branch method is of any help in
my context.

I think, this feature should be supported by the DSL as the Processor API
solution is not completely safe.


2016-10-18 10:01 GMT+02:00 Damian Guy <damian.guy@gmail.com>:

> Hi Florian,
>
> Do you know the set of subtypes in advance? I.e, could you use:
>
> KStream[] branches = stream.branch(predicates);
>
> to split the stream based on the subtypes?
>
> Thanks,
> Damian
>
>
> On Tue, 18 Oct 2016 at 00:43 Matthias J. Sax <matthias@confluent.io>
> wrote:
>
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA512
> >
> > Hi,
> >
> > using DSL you cannot do this. However, if you use Processor API you can.
> >
> > There are similar question on the mailing list already. For example:
> > http://search-hadoop.com/m/uyzND1lghNN1tzbf41&subj=kafka+stream+to+new+t
> > opic+based+on+message+key
> >
> > As we got this request multiple times already, it might be worth
> > adding it IMHO. Not sure what the opinion of other is? We should make
> > sure that the feature gets accepted before you put a lot of effort in
> > it. :)
> >
> >
> > - -Matthias
> >
> > On 10/17/16 2:10 PM, Florian Hussonnois wrote:
> > > Hi All,
> > >
> > > Currently, it seems not possible with KStream to produce messages
> > > to topics which are not known until runtime.
> > >
> > > For a new project I am evaluating the Kafka Connect / Kafka
> > > Streams architecture but without that feature I cannot retain the
> > > KStreams API.
> > >
> > > Our use case is pretty basic. We have xml messages in input of our
> > > topology. Each message is splitted into subtypes and formatted in
> > > Avro before being sent to a dedicated topic.
> > >
> > > So the output topics depend of the subtype of each message.
> > >
> > > I think it would be nice to add methods into the KStream interface
> > > to provide such feature.
> > >
> > > If you think that feature would be usefull I can create a jira and
> > > contribute to it. Also, do I need to create a new KIP as this
> > > requires changes on a public API ?
> > >
> > > Thanks,
> > >
> > -----BEGIN PGP SIGNATURE-----
> > Comment: GPGTools - https://gpgtools.org
> >
> > iQIcBAEBCgAGBQJYBWIhAAoJECnhiMLycopPfTQQAI69Uii5xd8KvaEo/Aeqs0Xw
> > AzdPHekdVoHANzo1h45W1x3/lnyeMU/n2v09Agsz46cxb+Xbz9NOKGqT3v9Ye0Ic
> > Eyl5yib1B6sWr41rGuYmwDH8zBoC8dPfGZiWhfXL4Sypey3RWzQlVAUWg8Ob4tqF
> > rFeubMjWp7yopKRe/7n//JHF029hVK/ePk1vdEsI+2lBI4N7q9ONT/1wKkeCAtdd
> > CCkI2WP/WbHzCcUVmOL41KoqgQFnmrH7BtLH67jumzEIR16H+ZenGZmS1uzde56E
> > 9mEsl4wmAvfB5GJu6y7JnS7FnQotw7pV7ZneQrA2q8eCZHZqs2fkXf+6ZJNYRir+
> > rysqt8wJG69ZN9bSNO1Q6/fNbRiSjYi0I7JnzkErP6scfDKlf3bWzlw6Ejc0+iUr
> > Cd0x2m/RlCepVleMT0UZNDlJd0Ml9Q77npP1lyntHVYHjVvtZLdlB5BQYdTMAx3N
> > KCLZ+WkY2CBKcwh/KuMr9kW2eWSxH89JSwEule+1bN4vSKyBA6vtrwDoshf6N23g
> > dEhTiY5NsgkvAe1JEK6d7PLN2Tq1Tq4OJNoP8PZlqW+YSFl41klQUblo8yT1jSlF
> > iCyQS4rgNRabjBs1iZnZNoZ5eodoJMcUyWPhHUYHne+MXuSr1cNNGeNcbS5W0UyE
> > dPCe2IiY4zErzxW/Mjmw
> > =4DpY
> > -----END PGP SIGNATURE-----
> >
>


-- 
Florian HUSSONNOIS

--089e0103e2f696092b053f249e65--