kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cyrus Vafadari <cy...@confluent.io>
Subject Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Date Tue, 04 Jun 2019 21:55:58 GMT
Randall,

I plan to update the public details section and the performance impact as
you recommended.

Regarding state-specific counts, I do agree this is a useful addition.
Before I make the change, I'd like to agree that these state-specific
counts should be in addition to the already-proposed total tasks count
(even though might be redundant, it is robust against new/missed connector
states, and is a useful metric in its own right), yes?

Cyrus

On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch <rhauch@gmail.com> wrote:

> Thanks, Cyrus -- this will be quite useful. I do have a few
> comments/requests.
>
> Can you please be more specific about the public details about the metric?
> What is the MBean name on which the metric will appear? For example, the AK
> documentation (https://kafka.apache.org/documentation/#connect_monitoring)
> defines all of the metrics an where they will appear, as does
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
> .
>
> Secondly, while a metric showing the total number of tasks is very useful,
> might it be worth considering also adding metrics for the number of running
> tasks, the number of paused tasks, and the number of failed tasks for a
> connector. It might require using the herder's `connectorStatus(String
> connectorName)` method instead, but that appears to be just as effective at
> using the local snapshot of the status store cache.
>
> Thirdly, it might be useful for the KIP to address the potential
> performance impact of computing these methods. Again, IIUC, the herder
> methods that the proposal mentions use the status and config stores caches
> only, so the impact should be negligible.
>
> Best regards,
>
> Randall
>
> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan <ryannedolan@gmail.com>
> wrote:
>
> > Cyrus, I agree this would be useful.
> >
> > Ryanne
> >
> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <odiachenko@apache.org
> >
> > wrote:
> >
> > >
> > >
> > > On 2019/05/30 06:06:12, Cyrus Vafadari <cyrus@confluent.io> wrote:
> > > > Hello Dev,
> > > >
> > > > I'd like to start the discussion of KIP-475: New Metric to Measure
> > Number
> > > > of Tasks on a Connector.
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
> > > >
> > > > The proposal is pretty straightforward -- to add a new metric to
> > Connect
> > > to
> > > > measure the number of tasks on a Connector. Currently, we support
> this
> > on
> > > > Worker level, so this KIP just adds another metric to support this
> > > > per-connector.
> > > >
> > > > There is also a PR:
> > > > https://github.com/apache/kafka/pull/6843
> > > >
> > > > Thanks,
> > > >
> > > > Cyrus
> > > >
> > >
> > > Hi Cyrus,
> > >
> > > That sounds like a useful addition.
> > >
> > > Regards, Alex.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message