From dev-return-100741-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Thu Jan 3 21:05:22 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8C85E180608 for ; Thu, 3 Jan 2019 21:05:21 +0100 (CET) Received: (qmail 13433 invoked by uid 500); 3 Jan 2019 20:05:20 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 13420 invoked by uid 99); 3 Jan 2019 20:05:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2019 20:05:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2B6FBD0FF7 for ; Thu, 3 Jan 2019 20:05:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.047 X-Spam-Level: ** X-Spam-Status: No, score=2.047 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=outlook.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 1JwX7cIZvv4u for ; Thu, 3 Jan 2019 20:05:16 +0000 (UTC) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-oln040092002038.outbound.protection.outlook.com [40.92.2.38]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id EA5EE5F576 for ; Thu, 3 Jan 2019 20:05:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SSWLrltyfcPj47XNePh8pOdbpjnLqtGQE2U/MwX7L7w=; b=k43QwwnSOn9VJUANqhJRm6Eu7WzWMIsYAzLFM6/8pOmZHiASXdXL4j1xtGl+Se9zmICbYZ0g6ieeOYHTWNp0kAgqvkDTgtV2IuCokCD2NztiCx7vRvaQAToi9uqh/F8BPgC1CqP/c+HjCfLNw9Ka2swGqbMBuzp252xZ+oo/He/oGIw6uu9ziIvVJsI6UnNgzwixrmuwNTD1oJuWnITxr1Mxe2NBKtRffJTp+gl2Dk3k1HcIV/W6hAkb9wKeL0k/iUF+0MG7HYchxsEcCH2UXCT9Jc5FBYn+Q9LOmazfJDMoI6yAMZICNS5vr3qSNxIy0V6oNBTk3msgUHFhbGAoWQ== Received: from SN1NAM01FT036.eop-nam01.prod.protection.outlook.com (10.152.64.59) by SN1NAM01HT045.eop-nam01.prod.protection.outlook.com (10.152.64.174) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1471.13; Thu, 3 Jan 2019 20:05:08 +0000 Received: from BYAPR10MB2615.namprd10.prod.outlook.com (10.152.64.56) by SN1NAM01FT036.mail.protection.outlook.com (10.152.64.120) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1471.13 via Frontend Transport; Thu, 3 Jan 2019 20:05:08 +0000 Received: from BYAPR10MB2615.namprd10.prod.outlook.com ([fe80::e1e5:ef16:5362:4968]) by BYAPR10MB2615.namprd10.prod.outlook.com ([fe80::e1e5:ef16:5362:4968%5]) with mapi id 15.20.1495.005; Thu, 3 Jan 2019 20:05:08 +0000 From: Boyang Chen To: "dev@kafka.apache.org" Subject: Re: Suggestion to make 0 grace period the default for suppress Thread-Topic: Suggestion to make 0 grace period the default for suppress Thread-Index: AQHUoNLh3XAXiWy64EygzRYkb6RdEaWd7/GAgAANJNU= Date: Thu, 3 Jan 2019 20:05:08 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:5967CD38066C1FEA20F108544BCD8A0F1D401E2AE4C880DC5CF71CD2C79EE95D;UpperCasedChecksum:D91615DEBBB8CEACF0C062AEA750663CABD8516F1DCDC9EE64B7746961538E53;SizeAsReceived:7152;Count:46 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [J5WTqiD9rd3OHFM6lohfGX4baxTxlnOc] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;SN1NAM01HT045;6:9kZv4AlxCTH8r4gBIQ3OqhGIkeg6bUyXYWOdarGFTYVg4hyNP0EyA8mBnjNCgWdiD9dgmfpy1TdPhdpIWcbM/HI5dJitAQ79Ctgmx2kThLWl73efAF0MTcWV8eOjbaw5rHow89/iArjac5gWkLAg0VWYw+jWUMl3sneU6t1NJ9IXmeq09OO3b+7YHha5fDtfBmZWpNBq/A6DSUjXp2CM6KK0Q5PFFJZI+Rr39cqVgCH1dlwdjtq2OBY7rR1Wwy1xpq4Wn/zEfM66kgKhdB/Q+x+8uZ6T/TCq9EyjysDj3W5Enxu9cB7qYYyvjBa+P1KJ1M3ItjscQi5HLBRx/K9vWpytzpH9XfVNJxegBM/EbOn0cMrR72QLfB6wO2tPnutwU9zG5NB0wze1DyCKc3217rubs6eKuCFlFznLqlebOQUjmsRNh8MpTaVr+S3Ja1jFVKrgP+WN7lWdXAnHBByplQ==;5:pH3nqhbwclAAu4b+uZn8D5rC8P2JvErJsVANDu1e1d2CVKhY50zpEs+IK8gLxSnAmmtpAOYeugIPoHKOSmMMQcg/ejEyz9GGlD/tBEqlslXwSfdP0bktMGAH7HRACD8K87jnWpp4LyqfZoPEQFPMfROoORsJCub4UDlfQkobpoQ=;7:5tNVFHpC257HYkaEktJa5yQ0I2caTM4Sc0528Q8pwfjOC2lOxYx/cExAZm3mds8WzhDFIkz8i3+f+TYcpAQnX+K7zL+iaz8wcKlIq2t9rOdzyY64Jv76oAavkMBQx72+bBhpzRhzcnWONtpxd+caJw== x-incomingheadercount: 46 x-eopattributedmessage: 0 x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(201702061078)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031323274)(2017031324274)(2017031322404)(1601125500)(1603101475)(1701031045);SRVR:SN1NAM01HT045; x-ms-traffictypediagnostic: SN1NAM01HT045: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(4566010)(82015058);SRVR:SN1NAM01HT045;BCL:0;PCL:0;RULEID:;SRVR:SN1NAM01HT045; x-microsoft-antispam-message-info: y5eR780xnGyADlbD0RRLUNez5vrFmv00puMuMA6FuSNEIeAMCSedV0Vbe60u9SLR Content-Type: multipart/alternative; boundary="_000_BYAPR10MB2615D13615B8B5507AABE827BE8D0BYAPR10MB2615namp_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 9bd8b953-1c55-4da7-b616-8bcad099ae8b X-MS-Exchange-CrossTenant-Network-Message-Id: b97e1488-4aff-4802-aff1-08d671b6c2f2 X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 9bd8b953-1c55-4da7-b616-8bcad099ae8b X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Jan 2019 20:05:08.7024 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1NAM01HT045 --_000_BYAPR10MB2615D13615B8B5507AABE827BE8D0BYAPR10MB2615namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thanks for the proposal Jingguo. Guozhang when you mention to "change the default value of retention to 0 consistently", will we introduce any backwa= rd incompatible issue since the retention cutoff might change on the changelog topics that are unexpect= ed. Boyang ________________________________ From: Guozhang Wang Sent: Friday, January 4, 2019 3:16 AM To: dev Subject: Re: Suggestion to make 0 grace period the default for suppress Thanks for reporting this Jingguo, personally I'd favor we change the default value of retention to 0 consistently, regardless of whether suppress is used, if we would ever consider changing it. Since it is a public API change (i.e. changing the default value), it should be discussed and voted via a KIP: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposa= ls Are you interested in creating one and drive the discussion? Guozhang On Sun, Dec 30, 2018 at 10:34 PM jingguo yao wrote: > [1] has the following code to demonstrate the usage of suppress method. > > KGroupedStream grouped =3D ...; > grouped > .windowedBy(TimeWindows.of(Duration.ofHours(1)).grace(ofMinutes(10))) > .count() > .suppress(Suppressed.untilWindowCloses(unbounded())) > .filter((windowedUserId, count) -> count < 3) > .toStream() > .foreach((windowedUserId, count) -> > sendAlert(windowedUserId.window(), windowedUserId.key(), count)); > > > If I remove the grace method invocation, I will have a one day > retention period. The following code in > org.apache.kafka.streams.kstream.TimeWindows causes this hehaviour: > > @SuppressWarnings("deprecation") // continuing to support > Windows#maintainMs/segmentInterval in fallback mode > @Override > public long gracePeriodMs() { > // NOTE: in the future, when we remove maintainMs, > // we should default the grace period to 24h to maintain the default > behavior, > // or we can default to (24h - size) if you want to be super accurate. > return grace !=3D null ? grace.toMillis() : maintainMs() - size(); > } > > I think that it is better to use 0 grace period if > "suppress(Suppressed.untilWindowCloses(unbounded()))" exists. With the > suppress method invocation, people are expecting to see the final > window result when the window closes instead of wait to see the result > after the one-day period. Even if we have some reasons similar to ones > mentioned in the code comment, it is better to mention this hehaviour > somewhere in Kafka streams documentation. > > > [1] > http://kafka.apache.org/21/documentation/streams/developer-guide/dsl-api.= html#window-final-resu > > > -- > Jingguo > -- -- Guozhang --_000_BYAPR10MB2615D13615B8B5507AABE827BE8D0BYAPR10MB2615namp_--