From user-return-24972-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Dec 19 11:45:41 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9EE70180670 for ; Wed, 19 Dec 2018 11:45:40 +0100 (CET) Received: (qmail 63019 invoked by uid 500); 19 Dec 2018 10:45:39 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 63008 invoked by uid 99); 19 Dec 2018 10:45:39 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Dec 2018 10:45:39 +0000 Received: from [10.0.1.20] (dslb-002-205-086-134.002.205.pools.vodafone-ip.de [2.205.86.134]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 1B8801050; Wed, 19 Dec 2018 10:45:37 +0000 (UTC) Subject: Re: Custom Metrics in Windowed AggregateFunction To: Chirag Dewan , User References: <175945353.5438044.1545190255412.ref@mail.yahoo.com> <175945353.5438044.1545190255412@mail.yahoo.com> From: Dawid Wysakowicz Openpgp: preference=signencrypt Autocrypt: addr=dwysakowicz@apache.org; keydata= mQINBFlDn4sBEADAXnPD90xKDWhCifMUi7CQhC2x6YAAnkIAosTqtJiwKQWxD6H/gJof8h6u WryDyvg9aQa6svDYHjnhFlJ15QNUZM6ssqfwtsFx/wPn4NqkOnEXWyybRzv4++Hdci/3fQTu Ht+r69fnDQN/yDCK22KYLBj5r9aqX/M8L3pehulJXHSM/2A5FR5UocWgOh+pbc5ZXwxr72oF DOo+xG31/sMwYV/rQw+6ab5zdNOcuKPR1vxxMGFmYkP1fn3LEiVAq4TGB3y9tpH3jof0u0oS UNRMZg3lQfJxcyhJk+R2Pu+sg3TYI2mAm0sOwxWtl5nn1qxkK8fNsXA9c4Mjc8QWqVxaIQtz yVAhlNQmcKEP0fQhU6m1f6CADaR23J6MHOswfQUbSvKHjmKSo+IjFAYW4PaofXrZwVJmB4KD rsIVnEMeVdYtR3PmABhf3h8ugFKOxQmBpX/CbcH2ycCehRNQhM1uCBy+IF2OGoINuSLCGpy2 ABt1DVGy3G4hgPzlI2OK3xGs7XJWlnO/1uDC0QeSsgr0QwgPJGY/+dumRbadtDd0eae4+/HY zH7vtQd9gXv6qcqVDZr8yd2fzgfo1JJxc6we+ujVwMN0NQkFw6DK0Rob9T5Ip/JfVfOlbIj8 SgIdZBNEHdtB3HEZRZOCGSYBM/8Opa4qXs+3AG3W4Z3vW49ogwARAQABtClEYXdpZCBXeXNh a293aWN6IDxkd3lzYWtvd2ljekBhcGFjaGUub3JnPokCPQQTAQoAJwUCWUOfqAIbAwUJB4Yf gAULCQgHAwUVCgkICwUWAgMBAAIeAQIXgAAKCRAx0t0Qv8FaLeukD/43Vk9Cc6+W76QeEG5o i0aCm4Z2B7Ddo+2Ha0Df6kp3LLYguMl04sU8B4u9+g64/+DZ+4CXYMBlMlmDFOj+k9OLIQTB rmR6CUsq1JVH91zGI3G/FzRWT1ZT5PAL65EJP+asKbjVZJEQIUokEvlzDWhhrwjMhybomC13 Tz76UWuEGXOx4LHadZsu/eaPfb4hZuXPL7e3/8NT+NSfE2BUSLt91Ez371SweU+8A0eT+iBn YG/gYvw19cmIg8CrC2bklrlfQqtMtxDsVXzzz2Ah13JFzQWRABJwA1oJwkrYsb/UvPU3DoSJ cWgzkhm0IKZIKH/YdgahuxF8pf8sp/rHBmJtqftFWDGtBFdmt4t1RXQpE0ikGC8Xa4fcO1Yq UKGPMWBW4DwSuVGbX0imsDC/D/ItvZ8FcjXyGKnCNanc+cZpKzkJpkxZvRsmU8jczSWhsuL1 3Fm9I39CLQtIKRJ0vCIGYK6jlBlNaYAxMNHY+V8GUjUMUcuA27zjOY/1XaOcAcATBBw3He2s QpHZutnAlxkFBoNw3Fs6UGRiurcZ8BXaFcdbdrNE+vv6a2zZA9zrPYPDGZrgsScC+8R8Jw/w 7+vXEQf/CSlALtbKdNsJm4NOLuU/PRqbSl0XWzi4tlV+vtueFVzfxFn55af5O+PrvGTa6tjr yozCG5oG9qOS5BR/9bkCDQRZQ5+LARAAt98V04resJ+wxz0uc8ME+K5fTJiIlbx4PttrMwqO CHbC6eoCnDlcHZE/iqv6tTcYJz8zrHiktpmmBYytL1x9a0lWtY8UjBy+3tjRal2RFCgoMo3X XC4Jlz8kc/4/p2ZQYFJKk35Y5R/Upngr2s7lHLFXoxWg6T9DMwEOoTl6ZC9JGDliyaA7sX56 LtHxkXJN/kEPspoeQenA14Mn2KakYv8ET3vD2HXWPk3EzkbyU0Wz7WOqkyYqNGMDvjGl+/Wl I3qqjrgvYnh7+WuDBpCsP9N2l3XncAwGiMWayePTsu/lkUemAe0LLR9ur5HU701rV9i3bp9U D2/z77VcN3kV3/KIDpMIPHGEGwFqnSW82KJ81fa/1YvZM245XXIPFfvTjJBq9+BqbP+Nvw6z S9pgrO8oo3Va4cgWL4lTKPOYzSWRLeiWUerFSOGXRu+zfu3zjwhy9hCp/DliPkDvJtSQi0MS Wum57ck8ReyIE6y8KKuYblvKXJVQcxaTckJpnx5g0VQKksGHSahD/C7aILp4rNQNkwjOzxh0 qsoBe8yq76XL97+ve7yALM5MUtLt/5LczkI/2EDuGkvXmtN8OpqCFCbqD8UK28bR3g71tfup 7iU6r02+27GTktRnMUpXQ2/u+xBZhKIEnYY9rHEL7lJj3FFntzdbO55K7qwOkn4Lo3EAEQEA AYkCJQQYAQoADwUCWUOfiwIbDAUJB4YfgAAKCRAx0t0Qv8FaLSzlD/9Xv0PaLcoPCeiKnhhn hQR9aLdatfAWBna+iVSDSkE+2g6XKzn6tBjh5J8SzMNVPfxyUjMCZjwW3kzbO6HOmQowrNYd gU4L7Wtn53kZipy1zRGvZAUwjwqs4KI4GUPIK3qrVdyDPAc8WO8wy4bzLsyMq5k1Q0Ud8RDF R03ABDVkElXIkPhe7tn8T9kvQ6apnsIvNs1IZJqBLtzp47f5bWfuSIdjQgVNnQgseeTg1Y08 dIDgvWfJBAuRrw3hdk3gLn13R8vvUL85jtyrk3/4zmivxrkEsllHwtmdobSE4LmJZu94uilD FcHAT0/Ykjmdj7WdnpYAcTRFxMmVgoStJHD/R6iETOB8doyO+L121xCPxJDWDNI95mIypn+i uVueGFmdDYC3V1nBCwuJ7EvVMdb4LmV2iDS00wLqJIfRcVbHnHiR+IhVVOdg3H4sDbw7kY55 SYLI8Ntf/qppwcL3VpvMxfNlL4Ejjam+fxsfA8Qxtvo8VCCzNoFJ8OpA+M8lMyjcmF29IGwK r1+/YDE5IzAfCmwU85KwBpHGwt+fvz6jPnX7fQlWjAbUknc5Z/yuP0UGg5+MAxJjxZgG1ZwS 2wGsUfwiE296v01fUAGnJ8yluUcPODX8ZxfHSSDUd56Dr/2w+pvx06Rjrd+paL0SsyaXqtqG HF857dd5zmwuhWur/g== Message-ID: <9ca78bb3-9a55-52ae-84fe-873e6f1ac681@apache.org> Date: Wed, 19 Dec 2018 11:45:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <175945353.5438044.1545190255412@mail.yahoo.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="FjGYUZbXkyPzitKUILzepKPiA29XPuEo5" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --FjGYUZbXkyPzitKUILzepKPiA29XPuEo5 Content-Type: multipart/mixed; boundary="uUdq6AThhvtKOurX98GfnmaU4QnFhIoNQ"; protected-headers="v1" From: Dawid Wysakowicz To: Chirag Dewan , User Message-ID: <9ca78bb3-9a55-52ae-84fe-873e6f1ac681@apache.org> Subject: Re: Custom Metrics in Windowed AggregateFunction References: <175945353.5438044.1545190255412.ref@mail.yahoo.com> <175945353.5438044.1545190255412@mail.yahoo.com> In-Reply-To: <175945353.5438044.1545190255412@mail.yahoo.com> --uUdq6AThhvtKOurX98GfnmaU4QnFhIoNQ Content-Type: multipart/alternative; boundary="------------81F1759021ED985D9A707CE6" Content-Language: en-US This is a multi-part message in MIME format. --------------81F1759021ED985D9A707CE6 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Chirag, I am afraid you are right you cannot access metrics from within AggregateFunction in WindowedStream. You can though use rich variant of WindowFunction, which is invoked for every window with the results of AggregateFunction. Would that be enough for your use case to use =2Eaggregate(aggregateFunction, windowFunction) and register metrics in the windowFunction? Best, Dawid On 19/12/2018 04:30, Chirag Dewan wrote: > Hi, > > I am writing a Flink job for aggregating events in a window.=C2=A0 > > I am trying to use the /AggregateFunction/ implementation for this.=C2=A0= > > Now, since WindowedStream does not allow a RichAggregateFunction for > aggregation, I cant use the RuntimeContext to get the Metric group.=C2=A0= > > I dont=C2=A0even see any other way of accessing the Metric Group in a > non-rich function implementation? > > Is there any way around here?=C2=A0 > > Any help appreciated. > > Thanks, > > Chirag --------------81F1759021ED985D9A707CE6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Hi Chirag,

I am afraid you are right you cannot access metrics from within AggregateFunction in WindowedStream. You can though use rich variant of WindowFunction, which is invoked for every window with the results of AggregateFunction. Would that be enough for your use case to use .aggregate(aggregateFunction, windowFunction) and register metrics in the windowFunction?

Best,

Dawid

On 19/12/2018 04:30, Chirag Dewan wrote:
Hi,

I am writing a Flink job for aggregating events in a window.=C2= =A0

I am trying to use the AggregateFunction implementation for this.=C2=A0

Now, since WindowedStream does not allow a RichAggregateFunction for aggregation, I cant use the RuntimeContext to get the Metric group.=C2=A0

I dont=C2=A0even see any other way of accessing the Metric Gro= up in a non-rich function implementation?

Is there any way around here?=C2=A0

Any help appreciated.

Thanks,

Chirag
--------------81F1759021ED985D9A707CE6-- --uUdq6AThhvtKOurX98GfnmaU4QnFhIoNQ-- --FjGYUZbXkyPzitKUILzepKPiA29XPuEo5 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE6pOkNbTiybTJ9TP2MdLdEL/BWi0FAlwaIVAACgkQMdLdEL/B Wi2TgRAAun22ywtqIgy9QDAz74DsPLZeRkzJPEa8KaAbvXrDWKbAS7i+JV/LL2OP 26DwkKTB0COLMS4tQPaeMiMLkSMpoWIqd+/G0Z7iJ2S+AlM3ElUhE6iYOuzPUuiC L5fpzdERlHUzpMXUMD8jO0H9DSdj4NPJmeNbiKTLTClYyfCj1TLU865ts2RuloS4 V7mwH90x1Z+SmcdtclQzDvG80vPPAHG6Psmp3xyGgm+Ksv93PLOdkeBIqUv8XWuH dmLRcUr7aOjDa88VpcZH1iQJTdmEXOjrQSBRh2SIiXf6fM1NyFYINDAihbHSWZWN FM1Klk46Fvq/Xywr5WAGl603d1MkxUtQnQHFyRz+KVgvZtG903RWlA3r1FWQ3UgM X5tlh71x+9l+cN3UqkW5daK2CGdamnaukmH7bZiXA8NuMKb7+4ZqhxK0U7gVv2pd EGKGJW7I+KkWn/S2UYQQFZZ1KgKfZMgDdj3ppqwDQIrb77uolE0TQyROiR/PhrpQ w+7FuvUBJXcBT8XBmgsu0aF3aa4U1sPz7P7+/ZiiBkUGNpj6cPsLLGDGvcuu8Iz6 zGsEJexGOCh2GFpKnpU94vkR2BCNSwZodgUiSWFcABcCqhhR/qF+mTPbJxxEUIZb Xx6+kXwIsSiCqEQLyMxx5x3M2r4rwtW97WC2gE2AYP+y9sRxf70= =j64b -----END PGP SIGNATURE----- --FjGYUZbXkyPzitKUILzepKPiA29XPuEo5--