Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Shivam Sharma <28shivamsharma@gmail.com>
Date: Fri, 24 Nov 2017 00:53:21 +0530
Message-ID: <CAPgL-yMowvZNRi+8kvtW5zXGjRa3iCVq-Db0RN3sJjx7mNXPAg@mail.gmail.com>
Subject: How & Where does flink stores data for aggregations.
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary="089e08234c488703df055eab5e3c"
archived-at: Thu, 23 Nov 2017 19:23:33 -0000

--089e08234c488703df055eab5e3c
Content-Type: text/plain; charset="UTF-8"

Hi All,

I have a small question regarding where does Flink stores data for doing
window aggregations. Lets say I am running following query on Flink table:

SELECT name, count(*)
FROM testTable
GROUP BY TUMBLE(rowtime, INTERVAL '1' MINUTE), name

So, If I understand above query properly so it must be saving data for 1
minute somewhere to find aggregations. If Flink is persisting this in
memory then my concern is if I increase interval to a DAY or more then it
will store the complete data for interval which can cross memory. If
persistence is disk then latency will be there.

Basically how do we solve such kind of use-cases using FLINK where
aggregation interval are quite high.

Thanks in advance

-- 
Shivam Sharma

--089e08234c488703df055eab5e3c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"color:#000000">Hi Al=
l,</div><div class=3D"gmail_default" style=3D"color:#000000"><br></div><div=
 class=3D"gmail_default" style=3D"color:#000000">I have a small question re=
garding where does Flink stores data for doing window aggregations. Lets sa=
y I am running following query on Flink table:</div><div class=3D"gmail_def=
ault" style=3D"color:#000000"><br></div><div class=3D"gmail_default" style=
=3D"color:#000000"><span style=3D"background-color:rgb(204,204,204)">SELECT=
 name, count(*)</span></div><div class=3D"gmail_default" style=3D"color:#00=
0000"><span style=3D"background-color:rgb(204,204,204)">FROM testTable</spa=
n></div><div class=3D"gmail_default" style=3D"color:#000000"><span style=3D=
"background-color:rgb(204,204,204)">GROUP BY TUMBLE(rowtime, INTERVAL &#39;=
1&#39; MINUTE), name</span></div><div class=3D"gmail_default" style=3D"colo=
r:#000000"><br></div><div class=3D"gmail_default" style=3D"color:#000000">S=
o, If I understand above query properly so it must be saving data for 1 min=
ute somewhere to find aggregations. If Flink is persisting this in memory t=
hen my concern is if I increase interval to a DAY or more then it will stor=
e the complete data for interval which can cross memory. If persistence is =
disk then latency will be there.</div><div class=3D"gmail_default" style=3D=
"color:#000000"><br></div><div class=3D"gmail_default" style=3D"color:#0000=
00">Basically how do we solve such kind of use-cases using FLINK where aggr=
egation interval are quite high.</div><div class=3D"gmail_default" style=3D=
"color:#000000"><br></div><div class=3D"gmail_default" style=3D"color:#0000=
00">Thanks in advance</div><div><br></div>-- <br><div class=3D"gmail_signat=
ure" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div dir=3D"ltr"><=
div dir=3D"ltr"><div dir=3D"ltr"><div style=3D"color:rgb(136,136,136)">Shiv=
am Sharma</div><div style=3D"color:rgb(136,136,136)"><br></div></div></div>=
</div></div></div>
</div>

--089e08234c488703df055eab5e3c--