flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Task manager memory configuration with intermediate results
Date Tue, 03 Feb 2015 09:48:14 GMT
Yes, I would really like to get rid of the distinction between operator and
network buffers.
Having all buffers been taken from the same pool is a good step towards
that goal. Until the assignment is dynamic, I prefer to have a config
option for the network / operator ratio.

+1 for the proposal and a ratio config option.

2015-02-03 9:52 GMT+01:00 Stephan Ewen <sewen@apache.org>:

> I like this approach and would suggest to make the ratio configurable. The
> default could be 50/50 or 60/40 (op heap / net heap)
>
> On Mon, Feb 2, 2015 at 6:45 PM, Ufuk Celebi <u.celebi@fu-berlin.de> wrote:
>
> > Currently, the memory configuration of a task manager encompasses two
> > things:
> >
> > 1) NETWORK buffers: Fixed amount of memory for the network buffer pool
> > (default: 2048 buffers, each 32 KB => 64 MB)
> >
> > 2) OPERATOR buffers: A configurable fraction of the available heap memory
> > (default 0.7) for the memory manager used by internal operators like sort
> > or hash
> >
> > With the recently added supported for intermediate results, intermediate
> > results live in the network stack and use buffers from the network buffer
> > pool. Currently, our memory management is not a problem, because we only
> > support ephemeral intermediate results, which are directly consumed in a
> > pipelined fashion (i.e. the buffer pools are short-lived).
> >
> > But with the upcoming support for persistent intermediate results (
> > https://github.com/apache/flink/pull/356) and fine-grained fault
> > tolerance, we need to rethink how we configure/divide the available
> memory
> > between the network stack and the operators as the network buffer pools
> > will live longer.
> >
> > I would suggest the following:
> >
> > 1) Remove the configuration for network buffers
> >
> > 2) Keep the fraction configuration, but internally divide it between
> > network stack and operators in a 50:50 fashion (in the future with
> dynamic
> > memory management, we will not even need to statically divide the memory
> > between network stack and operators).
> >
> >
> > What do you think? Is this reasonable? Should the division be
> configurable
> > as well?
> >
> > – Ufuk
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message