flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kruse, Sebastian" <Sebastian.Kr...@hpi.de>
Subject Hardware Requirements
Date Fri, 04 Jul 2014 13:46:25 GMT
Hi everyone,

I apologize in advance if that is not the right mailing list for my question. If there is
a better place for it, please let me know.

Basically, I wanted to ask if you have some statement about the hardware requirements of Flink
to process larger amounts of data beginning from, say, 20 GBs. Currently, I am facing issues
in my jobs, e.g., there are not enough buffers for safe execution of some operations. Since
the machines that run my TaskTrackers have unfortunately very limited main memory, I cannot
increase the number of buffers (and heap space in general) too much. Currently, I assigned
them 1.5 GB.

So, the exact questions are:

*         Do you have experiences with a suitable HW setup for crunching larger amounts of
data, maybe from the TU cluster?

*         Are there any configuration tips, you can provide, e.g. pertaining to the buffer
configuration?

*         Are there any general statements on the growth of Flink's memory requirements wrt.
to the size of the input data?

Thanks for your help!
Sebastian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message