kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Roesler <j...@confluent.io>
Subject Re: Topology with loops (intermediate topics) and potential bug in TopologyTestDriver
Date Wed, 11 Sep 2019 16:09:26 GMT
Hi Adam,

I haven't looked into your project yet, but just wanted to mention
this to see if it explains your observations.

TopologyTestDriver processes every input record fully and
synchronously. So, when you pipe an input record in, it traverses all
internal processing, including intermediate topics, until all results
are written to sink topics (or the end of the topology is reached).

This may indeed result in a different order of output records than
you'd observe with Kafka Streams, which would instead write to the
intermediate topic in one thread and then process it in a separate
thread.

Does that add up with what you're seeing?

Thanks,
-John

On Wed, Sep 11, 2019 at 6:40 AM Adam Domanski <adomanski@gmail.com> wrote:
>
> Hi,
>
> I probably found a bug in TopologyTestDriver for quite non-trivial Kafka
> Streams topology.
>
> The streaming logic is the following: There is a concept of children and
> parents. Children are aggregated under parent. Some children of master
> parent can send poison pills to other parents. Such parents die but already
> aggregated kids should go then to the master. Application remembers killed
> parents in a local store. If a new kid comes, the store is checked and then
> kid is rerouted to the master.
>
> I'm using intermediate topics in my topology as well as sometimes
> tombstones.
>
> Here is Gradle (Java 11 + Kotlin) based project which demonstrate the
> issue: https://github.com/czterocyty/kafka_streams_test_case
>
> You can see that few tests fail there as I expect that latest output Parent
> records have the latest generation.
>
> So seems that TopologyTestDriver does not care about the order of output
> records.
>
> Best regards,
> Adam DomaƄski

Mime
View raw message