From dev-return-100792-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Mon Jan 7 02:16:48 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3B442180636 for ; Mon, 7 Jan 2019 02:16:48 +0100 (CET) Received: (qmail 24748 invoked by uid 500); 7 Jan 2019 01:16:46 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 24723 invoked by uid 99); 7 Jan 2019 01:16:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jan 2019 01:16:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C20FC180EAA for ; Mon, 7 Jan 2019 01:16:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.654 X-Spam-Level: * X-Spam-Status: No, score=1.654 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.145, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=salesforce.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id UmY83g1HcGdS for ; Mon, 7 Jan 2019 01:16:43 +0000 (UTC) Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 3CFDB5F60E for ; Mon, 7 Jan 2019 01:16:43 +0000 (UTC) Received: by mail-wm1-f49.google.com with SMTP id y185so5024226wmd.1 for ; Sun, 06 Jan 2019 17:16:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=salesforce.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=YFKFsGbwG0785z/W736KCLVzkkb3mA96M8MdpiPdZAc=; b=f5X8F1xTnwUrDpaU6fcqExlG95K8FPiojd+A34pSPftCoC+B0y/Vjmq0iyZJc0Qrqm WMS0dNe09qIVmx9YFIY1Aa1qH/eioiJLo+ugf9J0np/tCtgwzuP4CedaFP2ZCZi0nVwk 0L8fqj9KMn9k8GnSMiQYgj+UHdf6BQAXJ4LAI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=YFKFsGbwG0785z/W736KCLVzkkb3mA96M8MdpiPdZAc=; b=UFBywt5Yvnoo5ovrpTSzWPAQvtWwgka/5gA0RVwkpCPvlbwIo0NusJxTfiFS++W26W gJLpQdLQ3YMJoTG6UjgNHE2vRjY1+hbU5Kdq9+6JhhYxKV68CaOd5Cd0ZnNlL8EdwPCR ypOccqNJdp3fJyvLzMWMLAUyX3yMU8zhJ0cPUfnZ82rij3ySiAOSRoS7usHzjdOr7xTR bb9gLItW/iZQQzrrtPbjuIKmS82Q5E6HIZtHM6wKUPr2IJJ1liW7ZHDq8LtoRlKgztSy O9YEFP7IoOKP8QdNJ/PY7L24r0E8FProTqGedKHkq01PtWqX0Ku1+TU9lAX/kzgcG/A0 HxFw== X-Gm-Message-State: AJcUukeDvDWdW2v8tZDnZex2bMaqqEMG2O3kK0SEGCt0OrcdPmAmidB3 HxEdhgupnXYPsi2N5BUL66d6rs9qvPZN4QMIlhFEf6O4 X-Google-Smtp-Source: ALg8bN5otmHM6lsJ8o7hcmlKFmh6TwtB9ZBsT1WcSHziku5ER0rc1z+N0YE1RHyb9syCr7WR1WsdxnWF1JOrU/tRfy0= X-Received: by 2002:a1c:2e43:: with SMTP id u64mr7500492wmu.52.1546823795291; Sun, 06 Jan 2019 17:16:35 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Paul Davidson Date: Sun, 6 Jan 2019 17:16:09 -0800 Message-ID: Subject: Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique To: dev@kafka.apache.org Content-Type: multipart/alternative; boundary="000000000000d97482057ed3faf9" --000000000000d97482057ed3faf9 Content-Type: text/plain; charset="UTF-8" Hi Ewen, Thanks for the feedback! If that's the case, would it maybe be possible to compatibly change the > default to use task IDs in the client ID, but only if we don't see an > existing override from the worker config? This would only change the > behavior when someone is using the default, but since the default would > just use what is effectively a random ID that is useless for monitoring > metrics, presumably this wouldn't affect any existing users. I think that > would avoid having to introduce the config, give better out of the box > behavior, and still be a safe, compatible change to make. Yes, it would be possible to simply change the default. There were two reasons I went with a config option in the proposal. I was primarily trying to avoid giving users a surprise by altering existing behavior. But, as you said, people are unlikely be relying on the current random default for monitoring so this might be fine. Another minor advantage of my proposal is it keeps open the option of setting arbitrary custom client IDs. If we only append the task ID to the default client ID then setting the "producer.clientid" or "consumer.clientid" will always result in a name conflict and will almost always need to be avoided. If a change in the default behavior is acceptable, and we're OK with advising people to avoid setting "producer.clientid" or "consumer.clientid" in future, then I'm happy to go with your suggested approach of changing the default. ... or change the type to be an > enum and make it something like task.client.ids=[default|task] and leave it > open for extension in the future if needed). If we were going with new config, I like the enum option. > *"Allow overriding client.id on a per-connector > basis"* > > > > This is a much more complex change, and would require individual > > connectors to be updated to support the change. In contrast, the proposed > > approach would immediately allow detailed consumer/producer monitoring > for > > all existing connectors. > > > I don't think this is quite accurate. I think the reason to reject is that > for your particular requirement for metrics, it simply doesn't give enough > granularity (there's only one value per entire connector), but it doesn't > ... > config value (as we now do with managed secrets). For example, it could > support something like client.id=connector-${taskId} and the task ID would > be substituted automatically into the string. > Yes, I think I was misinterpreting "on a per-connector basis" to mean giving each connector a way to override client properties on a per-task basis: a simple per-connector override clearly wouldn't work. I was imagining adding a way to intercept and modify the client properties before they're used by the Worker to construct the client. You're right that this wouldn't necessarily need to live in the connector. Config interpolation is an option, as you mentioned. Either way, this is certainly a more complex change. Paul --000000000000d97482057ed3faf9--