kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics
Date Thu, 29 Mar 2018 21:03:16 GMT
Thanks for the KIP John.

Reading the material on the related Jiras, I am wondering what `reason`
tags you want to introduce? Can you elaborate? The KIP should list those

About the fine grained metrics vs the roll-up: you say that

> the coarse metric aggregates across two dimensions simultaneously

Can you elaborate why this is an issue? I am not convinced atm that we
should put the fine grained metrics into INFO level and remove the
roll-up at thread level.

> Given that they have to do this sum to get a usable top-level view

This is a fair concern, but I don't share the conclusion. Offering a
built-in `KafkaStreams` "client" roll-up out of the box might be a
better solution. In the past we did not offer this due to performance
concerns, but we could allow an "opt-in" mechanism. If you disagree, can
you provide some reasoning and add them to the "Rejected alternatives"

To rephrase: I understand the issue about missing top-level view, but
instead of going more fine grained, we should consider to add this
top-level view and add/keep the fine grained metrics at DEBUG level only

I am +1 to add TopologyTestDriver#metrics() and to remove old metrics
directly as you suggested.


On 3/28/18 6:42 PM, Ted Yu wrote:
> Looks good to me.
> On Wed, Mar 28, 2018 at 3:11 PM, John Roesler <john@confluent.io> wrote:
>> Hello all,
>> I am proposing KIP-274 to improve the metrics around skipped records in
>> Streams.
>> Please find the details here:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 274%3A+Kafka+Streams+Skipped+Records+Metrics
>> Please let me know what you think!
>> Thanks,
>> -John

View raw message