kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathieu Fenniak <mathieu.fenn...@replicon.com>
Subject Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
Date Tue, 07 Feb 2017 14:32:58 GMT
On Mon, Feb 6, 2017 at 2:35 PM, Matthias J. Sax <matthias@confluent.io>
wrote:

> - adding KStreamBuilder#topologyBuilder() seems like be a good idea to
> address any concern with limited access to TopologyBuilder and DSL/PAPI
> mix-and-match approach. However, we should try to cover as much as
> possible with #process(), #transform() etc.
>

That sounds like it'll work for me.


> - about TopologyBuilder.nodeGroups & TopologyBuilder.build: not sure
> what analysis you do -- there is also KafkaStreams#toString() that
> describes the topology/DAG of the job. @Mathieu: Could you use this for
> your analysis?
>

Well, I'd like to be able to output a graphviz diagram of my processor
topology.  I am aware of KafkaStreams#toString(), but, it isn't the format
I want, if I remember correctly I found it was ambiguous to parse &
transform, and it also has the limitation of requiring a running and
processing application as toString() doesn't return anything useful until
the consumer stream threads are running.

What I've whipped up with the existing ProcessorTopology API (
https://gist.github.com/mfenniak/04f9c0bea8a1a2e0a747d678117df9f7) just
builds a "dry" topology (ie. no data being processed) and outputs a graph.
It's hooked into my app so that I can run with a specific command-line
option to output the graph without having to start the processor.

It's not the worst thing in the world to lose, or to have to jump through
some reflection hoops to do. :-)  Perhaps a better approach would be to
have an API designed specifically for this kind of introspection,
independent of the much more commonly used API to build a topology.

Mathieu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message