flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thanh-Nhan Vo <Thanh-Nhan...@bleckwen.ai>
Subject RE: [Flink 1.6] How to get current total number of processed events
Date Wed, 23 Jan 2019 17:37:38 GMT
Hi Kien Truong,

Thank you for your answer. I have another question, please !
If I count the number of messages processed for a given key j (denoted c_j), is there a way
to retrieve max{c_j}, min{c_j}?

Thanks

De : Kien Truong [mailto:duckientruong@gmail.com]
Envoyé : mercredi 23 janvier 2019 16:04
À : user@flink.apache.org
Objet : Re: [Flink 1.6] How to get current total number of processed events


Hi Nhan,

Logically, the total number of processed events before an event cannot be accurately calculated
unless events processing are synchronized.

This is not scalable, so naturally I don't think Flink supports it.

Although, I suppose you can get an approximate count by using a non-keyed TumblingWindow,
count the item inside the window, then use that value in the next window.



Regards,

Kien


On 1/21/2019 9:34 PM, Thanh-Nhan Vo wrote:
Hello all,

I have a question, please !
I'm using Flink 1.6 to process our data in streaming mode.
I wonder if at a given event, there is a way to get the current total number of processed
events (before this event).
If possible, I want to get this total number of processed events as a value state in Keystream.
It means that for a given key in KeyStream, I want to retrieve not only the total number of
processed events for this key but also the total number of processed events for all keys.

There is a way to do this in Flink 1.6, please!
Best regard,
Nhan


Mime
View raw message