kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cyrus Vafadari <cy...@confluent.io>
Subject Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector
Date Tue, 04 Jun 2019 23:34:54 GMT
Randall,

I've updated the KIP to include all of your recommendations!

Cyrus

On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari <cyrus@confluent.io> wrote:

> Randall,
>
> I plan to update the public details section and the performance impact as
> you recommended.
>
> Regarding state-specific counts, I do agree this is a useful addition.
> Before I make the change, I'd like to agree that these state-specific
> counts should be in addition to the already-proposed total tasks count
> (even though might be redundant, it is robust against new/missed connector
> states, and is a useful metric in its own right), yes?
>
> Cyrus
>
> On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch <rhauch@gmail.com> wrote:
>
>> Thanks, Cyrus -- this will be quite useful. I do have a few
>> comments/requests.
>>
>> Can you please be more specific about the public details about the metric?
>> What is the MBean name on which the metric will appear? For example, the
>> AK
>> documentation (https://kafka.apache.org/documentation/#connect_monitoring
>> )
>> defines all of the metrics an where they will appear, as does
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework
>> .
>>
>> Secondly, while a metric showing the total number of tasks is very useful,
>> might it be worth considering also adding metrics for the number of
>> running
>> tasks, the number of paused tasks, and the number of failed tasks for a
>> connector. It might require using the herder's `connectorStatus(String
>> connectorName)` method instead, but that appears to be just as effective
>> at
>> using the local snapshot of the status store cache.
>>
>> Thirdly, it might be useful for the KIP to address the potential
>> performance impact of computing these methods. Again, IIUC, the herder
>> methods that the proposal mentions use the status and config stores caches
>> only, so the impact should be negligible.
>>
>> Best regards,
>>
>> Randall
>>
>> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan <ryannedolan@gmail.com>
>> wrote:
>>
>> > Cyrus, I agree this would be useful.
>> >
>> > Ryanne
>> >
>> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko <
>> odiachenko@apache.org>
>> > wrote:
>> >
>> > >
>> > >
>> > > On 2019/05/30 06:06:12, Cyrus Vafadari <cyrus@confluent.io> wrote:
>> > > > Hello Dev,
>> > > >
>> > > > I'd like to start the discussion of KIP-475: New Metric to Measure
>> > Number
>> > > > of Tasks on a Connector.
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector
>> > > >
>> > > > The proposal is pretty straightforward -- to add a new metric to
>> > Connect
>> > > to
>> > > > measure the number of tasks on a Connector. Currently, we support
>> this
>> > on
>> > > > Worker level, so this KIP just adds another metric to support this
>> > > > per-connector.
>> > > >
>> > > > There is also a PR:
>> > > > https://github.com/apache/kafka/pull/6843
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Cyrus
>> > > >
>> > >
>> > > Hi Cyrus,
>> > >
>> > > That sounds like a useful addition.
>> > >
>> > > Regards, Alex.
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message