From dev-return-106130-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Mon Jul 29 21:48:10 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id B2EDC18063F for ; Mon, 29 Jul 2019 23:48:09 +0200 (CEST) Received: (qmail 34595 invoked by uid 500); 29 Jul 2019 21:48:07 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 34583 invoked by uid 99); 29 Jul 2019 21:48:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jul 2019 21:48:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D21A01A31F1 for ; Mon, 29 Jul 2019 21:48:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.801 X-Spam-Level: * X-Spam-Status: No, score=1.801 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=confluent.io Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id OLNb8WdKkciM for ; Mon, 29 Jul 2019 21:48:03 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::22e; helo=mail-oi1-x22e.google.com; envelope-from=jason@confluent.io; receiver= Received: from mail-oi1-x22e.google.com (mail-oi1-x22e.google.com [IPv6:2607:f8b0:4864:20::22e]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 904B17DC06 for ; Mon, 29 Jul 2019 21:48:02 +0000 (UTC) Received: by mail-oi1-x22e.google.com with SMTP id v186so46404565oie.5 for ; Mon, 29 Jul 2019 14:48:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=confluent.io; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=z501jG9waJVFeqaNridgoO6BiNQrus4CEtgnxTCpsTY=; b=UAZ0nDReI3KE+S+xpQvksvRF0rlm7Hn+3DPHf0VXHONGguB0TjoCoPzKh6q515nQOS SDk5iZlKoDtd4L6Q/IPaueaqNbjEF5jZO5BqPJ3LJSVYXkaqwH3eSbqyY0DKaMbNaGRq oDsuBy8iLMSbywwpdI4kr11G03q+GyWYNXw5iAMIUZTTe+J+1QB5vBiDKlyGESgc9H/8 HV57piDfU6o/lEnXZcJTDY2p5W1GnzIZelQ9kALXTPQ7bKsNwX/QA3as43r2ajHrpqcy ygD+A+LF0k1TG0wt7ZIfbTfgtx0wgYgOLAgRR45gFqcYJXVXQxEM50Hf20hfxyGg6Dqd 2AKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=z501jG9waJVFeqaNridgoO6BiNQrus4CEtgnxTCpsTY=; b=Vp9TNYx9u0h5p72ThnPoHKZKEH3NZ1Umu4CD13rb+ZMJBtJKCqPv2NP1jzveQX93ru RYzxWKCSwB2zp7gl8TgBQzWZ7a9UZ3+upQY5nCoN3zoklbvG58HZwbrAX6KELWV2UF2q jV9PxcbfDr6RxHttFsvOPqVArTbqy6lamdpxosf3yU0LV+RNpIq/dnBOtoFsCAmwxIif lQUWBUbKV/EN19SMb112nIe60kR9HGbGNmdzlBFX/7Ip9SyU9aTSnE9pWDupxV4tn33s aEBmh2SaJ7pFVRKwzNtIQ57V52Wph614w7VIo1w503/oxomeeohHlxSiu5GL+MwZ2MYY d48w== X-Gm-Message-State: APjAAAWvIXms6WkYozl9jx4PJO9LRsuGqsyYu+rE8yrPuJ3+2yIz8S84 M2CUwiBPi6l7W4+R/a2u8UEupRGhsDDCzEjccSxoavj+3WA= X-Google-Smtp-Source: APXvYqxunUfsmP/mUs1pV4d3bvG3RrsDv4QCB9GpYNFa7nlXwlxnaTr8hZw6utyzKjwyRmc5WU4oKjUYnFWTckk45OU= X-Received: by 2002:a05:6808:b02:: with SMTP id s2mr53663266oij.155.1564436880991; Mon, 29 Jul 2019 14:48:00 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jason Gustafson Date: Mon, 29 Jul 2019 14:47:50 -0700 Message-ID: Subject: Re: [DISCUSS] KIP-496: Administrative API to delete consumer offsets To: dev Content-Type: multipart/alternative; boundary="00000000000090c5f3058ed8d8e7" --00000000000090c5f3058ed8d8e7 Content-Type: text/plain; charset="UTF-8" Hi Guozhang, I have added metrics to the KIP. Please take a look. This gave me an excuse to also add a metric for the group rebalance rate, which probably would have made detecting KAFKA-8653 easier. Since this is a relatively straightforward KIP, I will go ahead and start a vote later this week if there are no further comments. Thanks, Jason On Mon, Jul 29, 2019 at 9:10 AM Guozhang Wang wrote: > Thanks for the replies Jason! > > 2. No I do not see any problems, just trying to understand how restrict we > are applying this rule :) Piggy-backing on the existing background thread > and check interval mechanism means we are not "eagerly" expiring either, > but I think this is fine. > > > Guozhang > > > On Thu, Jul 25, 2019 at 3:16 PM Jason Gustafson > wrote: > > > 1. Fixed, thanks! > > > > 2. Yes, that is what I was thinking. Do you see any problems? > > > > 3. Good point. Do you think a meter for expired and deleted offsets would > > be sufficient? > > > > 4. I considered it. I thought that might be a little dangerous for > dynamic > > groups which have subscriptions changing. If the first member to > discover a > > subscription change falls out, then offsets would be lost. Also, it > seemed > > a little more consistent with empty group expiration. From an offset > > expiration perspective, an empty group is just treated as a case where > the > > subscription is empty, which makes all offsets subject to expiration. > > > > > > Thanks, > > Jason > > > > On Thu, Jul 25, 2019 at 1:41 PM Guozhang Wang > wrote: > > > > > Hi Jason, > > > > > > Thanks for the KIP! I've made a pass on it and here are few comments: > > > > > > 1. " before the clients which ." --> incomplete sentence? > > > > > > 2. " Any committed offset for a partition which is not currently > > subscribed > > > to is subject to expiration." --> this may be an implementation detail, > > but > > > are we going to piggy-back on the same offsetsRetentionCheckIntervalMs > to > > > check for expirable offsets as well? > > > > > > Some meta comment: > > > > > > 3. Looking into the current broker-side metrics, we do not have a good > > user > > > visibility yet for offset removal either due to expiration or deletion, > > > maybe we should consider adding one? > > > > > > 4. Playing the devil's advocate here: for cases where deleting > expirable > > > offsets is needed (like you mentioned lag monitoring), should we just > > > by-pass the offset retention ms (by default it's one day) and remove > > > immediately? What scenarios would require those non-subscribed > partition > > > offsets to be retained longer? > > > > > > > > > Guozhang > > > > > > > > > On Tue, Jul 23, 2019 at 10:11 AM Jason Gustafson > > > wrote: > > > > > > > Hi All, > > > > > > > > I have a short KIP to add an api for consumer offset deletion: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-496%3A+Administrative+API+to+delete+consumer+offsets > > > > . > > > > Please take a look and let me know what you think. > > > > > > > > Thanks, > > > > Jason > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > -- > -- Guozhang > --00000000000090c5f3058ed8d8e7--