kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anastasia Vela <av...@confluent.io>
Subject Re: [DISCUSS] KIP-484: Expose metrics for group and transaction metadata loading duration
Date Tue, 02 Jul 2019 21:58:43 GMT
After further discussion with Anna, we decided the following:
- add the average metric, but noted in the KIP that the average value may
look low at times when there are many empty partitions that have a 0ms load
time
- set the window to 30sec, because there is no significance difference if
we set the window time to 3hrs, so I will keep the default value instead.

Thanks. Let me know any more concerns.
Anastasia

On Mon, Jul 1, 2019 at 9:13 AM Anastasia Vela <avela@confluent.io> wrote:

> Hey Gwen!
>
> Thanks for reviewing my KIP!
>
> 1. I did consider adding an Avg metric as well. Anna and I decided that a
> max would provide the crucial information. We just need to know if there
> was a long load time, and expose what that duration was so we understand
> there's downtime for such a long time. However I do agree that it may be
> necessary to compute averages if we want to give the max a reference point.
> I can easily add this if we believe it is necessary.
> 2. The default refers to the metric configuration set when you initialize
> KafkaServer. When I was running tests, the max value was computed over a
> window of 30 seconds, unless I changed the metrics config. So I noted that
> unless we change the config for this specific metric, it will be computed
> over the default window.
> 3. I proposed a 3 hour window because we have (very rarely) seen
> partitions take hours to load. 3 hours was an upper bound for how long a
> load could take. The way max works is that it computes the running max
> until the window has lapsed. Then it starts a new window and forgets the
> max value of the last window. So if a partition takes more than the window
> time to load, there will be one value in that window and the next load will
> be part of a new window. I guess it just depends on how we want it to be
> displayed on the graph. If it's ok for this behavior to happen, the window
> can be shrunk. Regarding the rate metric, I was actually thinking about
> doing this, but I was told that loads don't happen very often. But it is
> true that if the reload happens very often then that may be a problem.
>
> Thanks,
> Anastasia
>
> On Fri, Jun 28, 2019 at 4:27 PM Gwen Shapira <gwen@confluent.io> wrote:
>
>> Hey,
>>
>> Thank you for proposing this! Sounds really useful - we have
>> definitely seem some difficult to explain pauses in consumer activity
>> and this metric will let us correlate those.
>>
>> Few questions:
>> 1. Did you consider adding both Max and Avg metrics? Many of our
>> metrics have both (batch-size and message-size for example) and it
>> helps put the max value in context.
>> 2. You wrote: "Lengthening or shortening the 3 hour time window is up
>> for discussion (default is 30sec)."  and I'm not sure what default you
>> are referring to?
>> 3. Can you also give some background on why you are proposing 3h? I'm
>> guessing it is because loading the state from the topic happens rarely
>> enough that in 3h it will probably only happen once or not at all?
>> Perhaps we need a rate metric to see how often it actually happens (if
>> we have to reload offsets very often it is a different problem).
>>
>> Gwen
>>
>> On Tue, Jun 25, 2019 at 4:43 PM Anastasia Vela <avela@confluent.io>
>> wrote:
>> >
>> > Hi all,
>> >
>> > I'd like to discuss KIP-484:
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-484%3A+Expose+metrics+for+group+and+transaction+metadata+loading+duration
>> >
>> > Let me know what you think!
>> >
>> > Thanks,
>> > Anastasia
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message