kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: [VOTE] KIP-444: Refactor and Augment Metrics for Kafka Streams
Date Thu, 02 Apr 2020 21:02:21 GMT
Hello all,

While implementing the last piece of this KIP for the coming 2.6 release, I
realized that it is important to cover the following monitoring metrics as
well so I'd propose adding them as part of KIP-444 too:

Instance-level:

   - number of alive stream threads, INFO
   - number of alive cleanup threads, INFO
   - number of alive global threads, INFO
   - number of alive restore threads, INFO

Monitoring these numbers can help if any threads died unexpectedly while
the instance is still proceeding.

Thread-level:

   - avg / max number of records polled from the consumer per thread
   iteration, INFO
   - avg / max number of records processed by the task manager (i.e. across
   all tasks) per thread iteration, INFO

Ideally the all polled records can be processed as well within one
iteration --- if one observed either we polled too few records such that
thread is mostly idling, or polled too many records that the thread cannot
keep up, she should go ahead and tune the consumer configs.

Task-level:

   - number of current buffered records at the moment (i.e. it is just a
   dynamic gauge), DEBUG.

This is a finer grained metric indicating which task's processing cannot
keep up with the fetching throughput.


Please let me know if anyone has any concerns about the proposed metrics.


Guozhang



On Mon, Sep 9, 2019 at 5:17 PM Matthias J. Sax <matthias@confluent.io>
wrote:

> +1 (binding)
>
>
> -Matthias
>
> On 9/5/19 11:47 AM, Guozhang Wang wrote:
> > +1 from myself.
> >
> > I'm now officially closing this voting thread with the following tally:
> >
> > binding +1: 3 (Guozhang, Bill, Matthias voted on the DISCUSS thread).
> > non-binding +1: 2 (Bruno, John).
> >
> >
> > Guozhang
> >
> >
> > On Thu, Aug 22, 2019 at 8:16 AM Bill Bejeck <bbejeck@gmail.com> wrote:
> >
> >> +1 (binding)
> >>
> >> -Bill
> >>
> >> On Thu, Aug 22, 2019 at 10:55 AM John Roesler <john@confluent.io>
> wrote:
> >>
> >>> Hi Guozhang, thanks for cleaning this up.
> >>>
> >>> I'm +1 (non-binding)
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> On Thu, Aug 22, 2019 at 2:26 AM Bruno Cadonna <bruno@confluent.io>
> >> wrote:
> >>>
> >>>> Hi Guozhang,
> >>>>
> >>>> +1 (non-binding)
> >>>>
> >>>> Thank you for driving this!
> >>>> Bruno
> >>>>
> >>>> On Tue, Aug 20, 2019 at 8:29 PM Guozhang Wang <wangguoz@gmail.com>
> >>> wrote:
> >>>>>
> >>>>> Hello folks,
> >>>>>
> >>>>> I'd like to start a voting thread the following KIP to improve the
> >>> Kafka
> >>>>> Streams metrics mechanism to users. This includes 1) renaming changes
> >>> in
> >>>>> the public StreamsMetrics utils API, and 2) a major cleanup on the
> >>>> Streams'
> >>>>> own built-in metrics hierarchy.
> >>>>>
> >>>>> Details can be found here:
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams
> >>>>>
> >>>>> I'd love to hear your thoughts and feedbacks. Thanks!
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>
> >>>
> >>
> >
> >
>
>

-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message