flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivam Sharma <28shivamsha...@gmail.com>
Subject How & Where does flink stores data for aggregations.
Date Thu, 23 Nov 2017 19:23:21 GMT
Hi All,

I have a small question regarding where does Flink stores data for doing
window aggregations. Lets say I am running following query on Flink table:

SELECT name, count(*)
FROM testTable
GROUP BY TUMBLE(rowtime, INTERVAL '1' MINUTE), name

So, If I understand above query properly so it must be saving data for 1
minute somewhere to find aggregations. If Flink is persisting this in
memory then my concern is if I increase interval to a DAY or more then it
will store the complete data for interval which can cross memory. If
persistence is disk then latency will be there.

Basically how do we solve such kind of use-cases using FLINK where
aggregation interval are quite high.

Thanks in advance

-- 
Shivam Sharma

Mime
View raw message