flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishnu Viswanath <vishnu.viswanat...@gmail.com>
Subject Re: maximum size of window
Date Mon, 27 Jun 2016 23:25:22 GMT
Hi Kostas,

Thank you.
Yes 2) was exactly what I wanted to know.

- So if I am using RocksDB as state backend, does that mean that I don't
have to worry much about the memory available per node since RocksDB will
use RAM and Disk to store the window state?

Regards,
Vishnu


On Mon, Jun 27, 2016 at 6:19 PM, Kostas Kloudas <k.kloudas@data-artisans.com
> wrote:

> Hi Vishnu,
>
> I hope the following will help answer your question:
>
> 1) Elements are first split by key (apart from global windows) and then
> are put into windows. In other words, windows are keyed.
> 2) A window belonging to a certain key is handled by a single node. In
> other words, no matter how big the window is, its
>         state (the elements it contains) will never be split between two
> or more nodes.
> 3) Where the state is stored, depends on your state backend. Currently
> Flink supports an in-memory one, a filesystem one, and
>         a rocksDB one which is in the middle (first in-memory and then
> disk when needed). Of course you can implement your own.
>
> From the above, you can see that if you use the memory-backed state
> backend, then your window size is limited by the memory
> available at each of your nodes. If you use the fs state backend, then
> your state is stored on disk. Finally, rocksDB will initially
> use RAM and then spill on disk when no more memory is available.
>
> Here I have to add that the window documentation is currently being
> re-written to explain new features introduced in Flink 1.1,
> which include more flexible handling of late events and more explicit
> state garbage collection.
>
> So please stay tuned!
>
> I hope this helps at answering your question,
> Kostas
>
> > On Jun 27, 2016, at 8:22 PM, Vishnu Viswanath <
> vishnu.viswanath25@gmail.com> wrote:
> >
> > Hi All,
> >
> > - Is there any restriction on the size of a window in Flink with respect
> to the memory of the nodes?
> > - What happens if a window size grows more than size of a node, will it
> be split into multiple nodes?
> >
> > if I am going to have a huge window, should I have fewer nodes with more
> memory.
> > Is there any documentation on how memory is managed/handled in the case
> of windows and also in the case of joins.
> >
> > Regards,
> > Vishnu
>
>

Mime
View raw message