From user-return-45001-archive-asf-public=cust-asf.ponee.io@flink.apache.org  Tue Jul 27 07:15:26 2021
Return-Path: <user-return-45001-archive-asf-public=cust-asf.ponee.io@flink.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255])
	by mx-eu-01.ponee.io (Postfix) with ESMTPS id 83038180648
	for <archive-asf-public@cust-asf.ponee.io>; Tue, 27 Jul 2021 09:15:26 +0200 (CEST)
Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153])
	by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 3004E3ED56
	for <archive-asf-public@cust-asf.ponee.io>; Tue, 27 Jul 2021 07:15:21 +0000 (UTC)
Received: (qmail 87680 invoked by uid 500); 27 Jul 2021 07:15:18 -0000
Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@flink.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@flink.apache.org>
List-Post: <mailto:user@flink.apache.org>
List-Id: <user.flink.apache.org>
Delivered-To: mailing list user@flink.apache.org
Received: (qmail 87669 invoked by uid 99); 27 Jul 2021 07:15:18 -0000
Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2021 07:15:18 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 48F7CC0494
	for <user@flink.apache.org>; Tue, 27 Jul 2021 07:15:17 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org
X-Spam-Flag: NO
X-Spam-Score: -0.7
X-Spam-Level:
X-Spam-Status: No, score=-0.7 tagged_above=-999 required=6.31
	tests=[DKIMWL_WL_HIGH=-0.698, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
	DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2,
	RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled
Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new);
	dkim=pass (2048-bit key) header.d=apple.com
Received: from mx1-he-de.apache.org ([116.203.227.195])
	by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024)
	with ESMTP id Lqru3JECCP2F for <user@flink.apache.org>;
	Tue, 27 Jul 2021 07:15:16 +0000 (UTC)
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=17.171.2.68; helo=ma1-aaemail-dr-lapp02.apple.com; envelope-from=mason.chen@apple.com; receiver=<UNKNOWN> 
Received: from ma1-aaemail-dr-lapp02.apple.com (ma1-aaemail-dr-lapp02.apple.com [17.171.2.68])
	by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 7FAA07FAB1
	for <user@flink.apache.org>; Tue, 27 Jul 2021 07:15:16 +0000 (UTC)
Received: from pps.filterd (ma1-aaemail-dr-lapp02.apple.com [127.0.0.1])
	by ma1-aaemail-dr-lapp02.apple.com (8.16.0.42/8.16.0.42) with SMTP id 16R78GnQ010690;
	Tue, 27 Jul 2021 00:15:09 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id :
 content-type : mime-version : subject : date : in-reply-to : cc : to :
 references; s=20180706; bh=JNFAU136NNf+O97KQeC6At2992pLFKOM73A+axECkXI=;
 b=d1RhOjpM+8sv4uNQPoUZ2zgGXsw+eLiLCd5Xf1mym2rL17gBxgjychZmWbM9pmOItyqt
 z4T6hojA6IL4L+bBneShVAuApuGnbl4tigVB0RK/wirt1DPZT8ez7icX4Gp8ci+wHjTI
 qkJgs8SAvUF2Kg1W6EtHLillcyqHJPgM0GtPZRTe/Wo+D3KDE2RTdFTXA+hFejSH2Z0C
 QX5XvuhJWYCdjc74iuCiZXKJBV3JLwyRM1Vk/u1S5d12Ptev4Z+/1vUqTgF9CGVgcEM3
 p8+JPJUSUid/x+UDeRHn1gZBk7gSa3GYV33pckcvkKYO44XS3Gnye/CbMqGUvSkA2w4z HQ== 
Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149])
	by ma1-aaemail-dr-lapp02.apple.com with ESMTP id 3a236jxsm6-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO);
	Tue, 27 Jul 2021 00:15:09 -0700
Received: from rn-mailsvcp-mmp-lapp01.rno.apple.com
 (rn-mailsvcp-mmp-lapp01.rno.apple.com [17.179.253.14])
 by rn-mailsvcp-mta-lapp01.rno.apple.com
 (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15
 2021)) with ESMTPS id <0QWW00O816T8GSH0@rn-mailsvcp-mta-lapp01.rno.apple.com>;
 Tue, 27 Jul 2021 00:15:08 -0700 (PDT)
Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp01.rno.apple.com by
 rn-mailsvcp-mmp-lapp01.rno.apple.com
 (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15
 2021)) id <0QWW00A006K74100@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Tue,
 27 Jul 2021 00:15:08 -0700 (PDT)
X-Va-A: 
X-Va-T-CD: 50dec77c16ea765914bb101c427470bc
X-Va-E-CD: 9f6d79099b6a1505bb363729ce181723
X-Va-R-CD: 2b908c6240babf361c1c6d0b0e361b80
X-Va-CD: 0
X-Va-ID: b20f1f5c-cb98-457c-a7e1-572d34a914b2
X-V-A: 
X-V-T-CD: 50dec77c16ea765914bb101c427470bc
X-V-E-CD: 9f6d79099b6a1505bb363729ce181723
X-V-R-CD: 2b908c6240babf361c1c6d0b0e361b80
X-V-CD: 0
X-V-ID: bfb0e835-e75c-45ee-8189-e85401fe5ec4
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790
 definitions=2021-07-27_05:2021-07-27,2021-07-27 signatures=0
Received: from smtpclient.apple (unknown [17.149.208.163])
 by rn-mailsvcp-mmp-lapp01.rno.apple.com
 (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15
 2021))
 with ESMTPSA id <0QWW00WHH6T8RS00@rn-mailsvcp-mmp-lapp01.rno.apple.com>; Tue,
 27 Jul 2021 00:15:08 -0700 (PDT)
From: Mason Chen <mason.chen@apple.com>
Message-id: <095B76A7-6CE6-4D23-BE5F-CFA25D74DB8F@apple.com>
Content-type: multipart/alternative;
 boundary="Apple-Mail=_A4EACEDD-C81C-42AF-A3A0-B7290717F66E"
MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\))
Subject: Re: as-variable configuration for state ac
Date: Tue, 27 Jul 2021 00:15:06 -0700
In-reply-to: 
 <SYAPR01MB23035F63376FE2BF4E404F6EDAE99@SYAPR01MB2303.ausprd01.prod.outlook.com>
Cc: Mason Chen <mas.chen6345@gmail.com>,
        "user@flink.apache.org" <user@flink.apache.org>
To: Yun Tang <myasuka@live.com>
References: 
 <CAOu4dyeZGYOawmt57Q5iRw1kbg9VLmLbmi7hQD9ymBfmRd0TYQ@mail.gmail.com>
 <SYAPR01MB23035F63376FE2BF4E404F6EDAE99@SYAPR01MB2303.ausprd01.prod.outlook.com>
X-Mailer: Apple Mail (2.3654.120.0.1.13)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790
 definitions=2021-07-27_05:2021-07-27,2021-07-27 signatures=0


--Apple-Mail=_A4EACEDD-C81C-42AF-A3A0-B7290717F66E
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Yup, your understand is correct=E2=80=94that was the analogy I was =
trying to make!

> On Jul 26, 2021, at 7:57 PM, Yun Tang <myasuka@live.com> wrote:
>=20
> Hi Mason,
>=20
> In rocksDB, one state is corresponding to a column family and we could =
aggregate all RocksDB native metrics per column family. If my =
understanding is right, are you hoping that all state latency metrics =
for a particular state could be aggregated per state level?=20
>=20
>=20
> Best
> Yun Tang
> From: Mason Chen <mas.chen6345@gmail.com>
> Sent: Tuesday, July 27, 2021 4:24
> To: user@flink.apache.org <user@flink.apache.org>
> Subject: as-variable configuration for state ac
> =20
> We have been using the state backend latency tracking metrics from =
Flink 1.13. To make metrics aggregation easier, could there be a config =
to expose something like =
`state.backend.rocksdb.metrics.column-family-as-variable` that rocksdb =
provides to do aggregation across column families.
>=20
> In this case, it would be the various components of state.
>=20
> =
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployme=
nt/config/#state-backend-rocksdb-metrics-column-family-as-variable =
<https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deploym=
ent/config/#state-backend-rocksdb-metrics-column-family-as-variable>

--Apple-Mail=_A4EACEDD-C81C-42AF-A3A0-B7290717F66E
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">Yup, =
your understand is correct=E2=80=94that was the analogy I was trying to =
make!<br class=3D""><div><br class=3D""><blockquote type=3D"cite" =
class=3D""><div class=3D"">On Jul 26, 2021, at 7:57 PM, Yun Tang &lt;<a =
href=3D"mailto:myasuka@live.com" class=3D"">myasuka@live.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><meta =
charset=3D"UTF-8" class=3D""><div style=3D"font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; font-family: Calibri, Helvetica, sans-serif; =
font-size: 12pt; background-color: rgb(255, 255, 255);" class=3D"">Hi =
Mason,</div><div style=3D"font-style: normal; font-variant-caps: normal; =
font-weight: normal; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; =
background-color: rgb(255, 255, 255);" class=3D""><br =
class=3D""></div><div style=3D"font-style: normal; font-variant-caps: =
normal; font-weight: normal; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; =
background-color: rgb(255, 255, 255);" class=3D"">In rocksDB, one state =
is&nbsp;corresponding to a column family and we could aggregate all =
RocksDB native metrics per column family. If my understanding is right, =
are you hoping that all state latency metrics for a particular state =
could be aggregated per state level?&nbsp;</div><div style=3D"font-style: =
normal; font-variant-caps: normal; font-weight: normal; letter-spacing: =
normal; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; font-family: Calibri, Helvetica, sans-serif; =
font-size: 12pt; background-color: rgb(255, 255, 255);" class=3D""><br =
class=3D""></div><div style=3D"font-style: normal; font-variant-caps: =
normal; font-weight: normal; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; =
background-color: rgb(255, 255, 255);" class=3D""><br =
class=3D""></div><div style=3D"font-style: normal; font-variant-caps: =
normal; font-weight: normal; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; =
background-color: rgb(255, 255, 255);" class=3D"">Best</div><div =
style=3D"font-style: normal; font-variant-caps: normal; font-weight: =
normal; letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; font-family: =
Calibri, Helvetica, sans-serif; font-size: 12pt; background-color: =
rgb(255, 255, 255);" class=3D"">Yun Tang</div><div id=3D"appendonsend" =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: =
normal; letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none;" =
class=3D""></div><hr tabindex=3D"-1" style=3D"caret-color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; display: inline-block; width: 656.59375px;" =
class=3D""><span style=3D"caret-color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: =
normal; font-weight: normal; letter-spacing: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: =
none; float: none; display: inline !important;" class=3D""></span><div =
id=3D"divRplyFwdMsg" dir=3D"ltr" style=3D"caret-color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none;" class=3D""><font face=3D"Calibri, sans-serif" =
style=3D"font-size: 11pt;" class=3D""><b class=3D"">From:</b><span =
class=3D"Apple-converted-space">&nbsp;</span>Mason Chen &lt;<a =
href=3D"mailto:mas.chen6345@gmail.com" =
class=3D"">mas.chen6345@gmail.com</a>&gt;<br class=3D""><b =
class=3D"">Sent:</b><span =
class=3D"Apple-converted-space">&nbsp;</span>Tuesday, July 27, 2021 =
4:24<br class=3D""><b class=3D"">To:</b><span =
class=3D"Apple-converted-space">&nbsp;</span><a =
href=3D"mailto:user@flink.apache.org" class=3D"">user@flink.apache.org</a>=
 &lt;<a href=3D"mailto:user@flink.apache.org" =
class=3D"">user@flink.apache.org</a>&gt;<br class=3D""><b =
class=3D"">Subject:</b><span =
class=3D"Apple-converted-space">&nbsp;</span>as-variable configuration =
for state ac</font><div class=3D"">&nbsp;</div></div><div =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: =
normal; letter-spacing: normal; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><div =
dir=3D"ltr" class=3D""><div class=3D"">We have been using the state =
backend latency tracking metrics from Flink 1.13. To make metrics =
aggregation easier, could there be a config to expose something =
like&nbsp;`state.backend.rocksdb.metrics.column-family-as-variable` that =
rocksdb provides to do aggregation across column families.</div><div =
class=3D""><br class=3D""></div><div class=3D"">In this case, it =
would&nbsp;be the various components of state.</div><div class=3D""><br =
class=3D""></div><div class=3D""><a =
href=3D"https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/=
deployment/config/#state-backend-rocksdb-metrics-column-family-as-variable=
" =
class=3D"">https://ci.apache.org/projects/flink/flink-docs-release-1.13/do=
cs/deployment/config/#state-backend-rocksdb-metrics-column-family-as-varia=
ble</a></div></div></div></div></blockquote></div><br =
class=3D""></body></html>=

--Apple-Mail=_A4EACEDD-C81C-42AF-A3A0-B7290717F66E--