From dev-return-104735-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Wed Jun 5 04:02:20 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A748718064D for ; Wed, 5 Jun 2019 06:02:19 +0200 (CEST) Received: (qmail 69147 invoked by uid 500); 5 Jun 2019 04:02:16 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 69133 invoked by uid 99); 5 Jun 2019 04:02:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jun 2019 04:02:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 91733C60E4 for ; Wed, 5 Jun 2019 04:02:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.802 X-Spam-Level: * X-Spam-Status: No, score=1.802 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, TVD_PH_BODY_ACCOUNTS_PRE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 3a_I-OapU2Op for ; Wed, 5 Jun 2019 04:02:10 +0000 (UTC) Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 932005F19A for ; Wed, 5 Jun 2019 04:02:09 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id u17so14033730pfn.7 for ; Tue, 04 Jun 2019 21:02:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=bSyzbWDNW2AzH2u9fXPd2hT8WzeFYh40jiiajUhf3Z4=; b=BX7MTbkqB1ypJ4r1c/0Aajma6Be9Pe65DUQ6uyCQrxincVs5aS3+YZhPMQZ4GViWco gPcTI4F8M1G0aCmJ7+fUOX5nVFflpLy6CGBSldWTkGi3eHGcQQ8O/NRFKyMXzHy6ZpPG E01LPERJElnctU73VR/hxeH0k4SFQM5PsHfducl4XLpLLMDlnlKfEPVpGT7BcTnPsbkM eVyc0B5BoFfuB5TuTpBnJxKmaVTf5a8McQqX3Y2RTowzuRF8XaqbAsh5tdV5A7DeSZvF 4/oIsd9ONk6pk8k6JmXAEZp5s84PoIZqPiObk3EZTLF0/3V3Ad1rD/kdktW7+GD4Gw3v I3zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=bSyzbWDNW2AzH2u9fXPd2hT8WzeFYh40jiiajUhf3Z4=; b=Ms3g1LMNLRKjG7VeJKt9Ty8bO3obtNNTTVj+91m8cn9IixgKX1DSX3Qghoi77ip+Jv v2BBXcpo/kaPMBn+p8Bv4i6eNXdze8cnlfuNeR45FX+mlZ6ssbpZ+EVWZndInQF37xhQ zUJsi01iR3xcteLJFULqswrSQkq14k4JfX5ouw1vUfG4ePT6dbIHYpgX26DWa6xsmndy wIVvATzRilIi51f4/sTj0ZjrFzedrpQU+EkKJQF3dHhUDYJUwlPdt2KK5hT4DxUgUlyB BFEGAGXNyItqf2Bgdk2TEatCOWE1McfshuM+w6Ot3uFWiQlWmcVMjetDcwzjew36E30m cgCw== X-Gm-Message-State: APjAAAXA400J73+4oi3V9hNkb8apgCLyZHSG/2TWL2wTQGUGrn/0pZ/o mEWs+fd0T6R1CwFWXFFw93twC00Ix/x5H2mB11S5UQ== X-Google-Smtp-Source: APXvYqxFB7EdQgDYeICkHxR7SWMpAfsCK9tpOKDhFc2NNIRU5hz63dH5jf/TYKKbyZcwMoHLtso4rqXKy3R7lgPK3jU= X-Received: by 2002:a17:90a:6505:: with SMTP id i5mr38231305pjj.13.1559707322604; Tue, 04 Jun 2019 21:02:02 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Randall Hauch Date: Tue, 4 Jun 2019 23:03:04 -0500 Message-ID: Subject: Re: [DISCUSS] KIP-475: New Metric to Measure Number of Tasks on a Connector To: dev Content-Type: multipart/alternative; boundary="000000000000eae287058a8ba818" --000000000000eae287058a8ba818 Content-Type: text/plain; charset="UTF-8" Sorry that I was not clear. Yes, I was suggesting the state-specific counts were in addition to the simple task count you originally proposed. Thanks for taking my suggestion into account -- the updated KIP looks great. Thanks for contributing this improvement, Cyrus! Best regards, Randall On Tue, Jun 4, 2019 at 6:35 PM Cyrus Vafadari wrote: > Randall, > > I've updated the KIP to include all of your recommendations! > > Cyrus > > On Tue, Jun 4, 2019 at 2:55 PM Cyrus Vafadari wrote: > > > Randall, > > > > I plan to update the public details section and the performance impact as > > you recommended. > > > > Regarding state-specific counts, I do agree this is a useful addition. > > Before I make the change, I'd like to agree that these state-specific > > counts should be in addition to the already-proposed total tasks count > > (even though might be redundant, it is robust against new/missed > connector > > states, and is a useful metric in its own right), yes? > > > > Cyrus > > > > On Tue, Jun 4, 2019 at 12:24 PM Randall Hauch wrote: > > > >> Thanks, Cyrus -- this will be quite useful. I do have a few > >> comments/requests. > >> > >> Can you please be more specific about the public details about the > metric? > >> What is the MBean name on which the metric will appear? For example, the > >> AK > >> documentation ( > https://kafka.apache.org/documentation/#connect_monitoring > >> ) > >> defines all of the metrics an where they will appear, as does > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework > >> . > >> > >> Secondly, while a metric showing the total number of tasks is very > useful, > >> might it be worth considering also adding metrics for the number of > >> running > >> tasks, the number of paused tasks, and the number of failed tasks for a > >> connector. It might require using the herder's `connectorStatus(String > >> connectorName)` method instead, but that appears to be just as effective > >> at > >> using the local snapshot of the status store cache. > >> > >> Thirdly, it might be useful for the KIP to address the potential > >> performance impact of computing these methods. Again, IIUC, the herder > >> methods that the proposal mentions use the status and config stores > caches > >> only, so the impact should be negligible. > >> > >> Best regards, > >> > >> Randall > >> > >> On Sun, Jun 2, 2019 at 10:05 PM Ryanne Dolan > >> wrote: > >> > >> > Cyrus, I agree this would be useful. > >> > > >> > Ryanne > >> > > >> > On Fri, May 31, 2019, 7:10 PM Oleksandr Diachenko < > >> odiachenko@apache.org> > >> > wrote: > >> > > >> > > > >> > > > >> > > On 2019/05/30 06:06:12, Cyrus Vafadari wrote: > >> > > > Hello Dev, > >> > > > > >> > > > I'd like to start the discussion of KIP-475: New Metric to Measure > >> > Number > >> > > > of Tasks on a Connector. > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-475%3A+New+Metric+to+Measure+Number+of+Tasks+on+a+Connector > >> > > > > >> > > > The proposal is pretty straightforward -- to add a new metric to > >> > Connect > >> > > to > >> > > > measure the number of tasks on a Connector. Currently, we support > >> this > >> > on > >> > > > Worker level, so this KIP just adds another metric to support this > >> > > > per-connector. > >> > > > > >> > > > There is also a PR: > >> > > > https://github.com/apache/kafka/pull/6843 > >> > > > > >> > > > Thanks, > >> > > > > >> > > > Cyrus > >> > > > > >> > > > >> > > Hi Cyrus, > >> > > > >> > > That sounds like a useful addition. > >> > > > >> > > Regards, Alex. > >> > > > >> > > >> > > > --000000000000eae287058a8ba818--