From dev-return-104730-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Tue Jun 4 23:35:13 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7697B18064D for ; Wed, 5 Jun 2019 01:35:13 +0200 (CEST) Received: (qmail 10711 invoked by uid 500); 4 Jun 2019 23:35:11 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 10697 invoked by uid 99); 4 Jun 2019 23:35:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jun 2019 23:35:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id C61081A4515 for ; Tue, 4 Jun 2019 23:35:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.801 X-Spam-Level: * X-Spam-Status: No, score=1.801 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=confluent.io Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id RVvMjUbenm6Q for ; Tue, 4 Jun 2019 23:35:07 +0000 (UTC) Received: from mail-vs1-f42.google.com (mail-vs1-f42.google.com [209.85.217.42]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 173A35F17E for ; Tue, 4 Jun 2019 23:35:07 +0000 (UTC) Received: by mail-vs1-f42.google.com with SMTP id n2so6261066vso.6 for ; Tue, 04 Jun 2019 16:35:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=confluent.io; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=MDotx+1oRN3NJ9xD/OPs4DxQHzWzq4ikqsZClR9tx8A=; b=EC+1NmY7DZdokw8YBcSZU4X8QSnut1ddT8mS0KCINKfLuE1UbJ2VZ426gcRdiYEZbm MrcrNaekOjTLSRxE/zvnGqB/NNJ/3jXAZYf/bkxCLWFjHKIprjE0yqps3lNv3fsfnhQ7 vtI2L2A13Hva1CIYO9GsEusxcnLXqPwV1J1FWm7Zl1C79ln/SI2G3crgYjdXKa6qgoF9 Bato1KTgvbcOwllA1D+aLzeCNJZ/5TDIAEn9v0kXk9XliJYT8IstaAi4R6rqLsxI2vyQ jfv0f7Ec9Fkx/O44EbEGu22DJOZFGfffvmLyJEyJyUMBQpgS2hemGILXlNVywYBpe709 DQXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=MDotx+1oRN3NJ9xD/OPs4DxQHzWzq4ikqsZClR9tx8A=; b=d+nc0YbCU0Tmi3jDTXwpPymYbfenvXVW2I3YaxqLF8q1TS1KqhV5iIhhzJsCuf3R/7 AE7g/BSPPZ6zBQ0rOFXtV5rUK+QVYSoFPospHgK7szVVVinDCi/bro7Z1czXwoImlk01 l+ay5xRDnVGMwCZpdxuGA1zljG0EdhNYvMzbVh8uKU0Lld/9RYfPYkViAE4Yldqn3cdC G9s/G4ech22fSRfh/mPcbTNGqeYkb9TPuDsD6hEomboWurvwra9JGSllDrT4If6TmINO Ngsz5EAnhJzXAGFlYf31xNVdMXnoPLu79N1ehBwRe8oi19CNvv79Zzh5QBx7JGFju/RH aKqA== X-Gm-Message-State: APjAAAXKUCzvU2G/mzwAPFKy1W/IpYWGAWLjODWlKg7/MN82L9MPawcu zNJ4CA5S+0nc2zQCZXmp+mlXW92ddaADAxqIWlg4q5xwwKc= X-Google-Smtp-Source: APXvYqxMEMPIoAURDsvm7WlWZrr790q4vTTeu42azwix/PwjfKijGhdfcOsVXgTF/f2rOQ2E4UkLaZBKI+U7IIn99qA= X-Received: by 2002:a67:e905:: with SMTP id c5mr17896214vso.97.1559691305726; Tue, 04 Jun 2019 16:35:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Cyrus Vafadari Date: Tue, 4 Jun 2019 16:34:54 -0700 Message-ID: Subject: Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector To: dev@kafka.apache.org Content-Type: multipart/alternative; boundary="0000000000003cd52c058a87ee61" --0000000000003cd52c058a87ee61 Content-Type: text/plain; charset="UTF-8" Randall, I've updated the KIP to include all of your recommendations! Cyrus On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari wrote: > Randall, > > I plan to update the public details section and the performance impact as > you recommended. > > Regarding state-specific counts, I do agree this is a useful addition. > Before I make the change, I'd like to agree that these state-specific > counts should be in addition to the already-proposed total tasks count > (even though might be redundant, it is robust against new/missed connector > states, and is a useful metric in its own right), yes? > > Cyrus > > On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch wrote: > >> Thanks, Cyrus -- this will be quite useful. I do have a few >> comments/requests. >> >> Can you please be more specific about the public details about the metric? >> What is the MBean name on which the metric will appear? For example, the >> AK >> documentation (https://kafka.apache.org/documentation/#connect_monitoring >> ) >> defines all of the metrics an where they will appear, as does >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework >> . >> >> Secondly, while a metric showing the total number of tasks is very useful, >> might it be worth considering also adding metrics for the number of >> running >> tasks, the number of paused tasks, and the number of failed tasks for a >> connector. It might require using the herder's `connectorStatus(String >> connectorName)` method instead, but that appears to be just as effective >> at >> using the local snapshot of the status store cache. >> >> Thirdly, it might be useful for the KIP to address the potential >> performance impact of computing these methods. Again, IIUC, the herder >> methods that the proposal mentions use the status and config stores caches >> only, so the impact should be negligible. >> >> Best regards, >> >> Randall >> >> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan >> wrote: >> >> > Cyrus, I agree this would be useful. >> > >> > Ryanne >> > >> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko < >> odiachenko@apache.org> >> > wrote: >> > >> > > >> > > >> > > On 2019/05/30 06:06:12, Cyrus Vafadari wrote: >> > > > Hello Dev, >> > > > >> > > > I'd like to start the discussion of KIP-475: New Metric to Measure >> > Number >> > > > of Tasks on a Connector. >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector >> > > > >> > > > The proposal is pretty straightforward -- to add a new metric to >> > Connect >> > > to >> > > > measure the number of tasks on a Connector. Currently, we support >> this >> > on >> > > > Worker level, so this KIP just adds another metric to support this >> > > > per-connector. >> > > > >> > > > There is also a PR: >> > > > https://github.com/apache/kafka/pull/6843 >> > > > >> > > > Thanks, >> > > > >> > > > Cyrus >> > > > >> > > >> > > Hi Cyrus, >> > > >> > > That sounds like a useful addition. >> > > >> > > Regards, Alex. >> > > >> > >> > --0000000000003cd52c058a87ee61--