Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <D3EB4E45.3E695%roshan@hortonworks.com>
References: <D3EB4E45.3E695%roshan@hortonworks.com>
From: Stephan Ewen <sewen@apache.org>
Date: Wed, 31 Aug 2016 11:51:32 +0200
Message-ID: <CANC1h_v9y7gB+9wq_7zONCBGpB_BXTK7bDwhw5SzjE_sHqq14A@mail.gmail.com>
Subject: Re: Streaming - memory management
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a11446af8cdf3f5053b5b0af3
archived-at: Wed, 31 Aug 2016 09:51:35 -0000

--001a11446af8cdf3f5053b5b0af3
Content-Type: text/plain; charset=UTF-8

In streaming, memory is mainly needed for state (key/value state). The
exact representation depends on the chosen StateBackend.

State is explicitly released: For windows, state is cleaned up
automatically (firing / expiry), for user-defined state, keys have to be
explicitly cleared (clear() method) or in the future will have the option
to expire.

The heavy work horse for streaming state is currently RocksDB, which
internally uses native (off-heap) memory to keep the data.

Does that help?

Stephan


On Tue, Aug 30, 2016 at 11:52 PM, Roshan Naik <roshan@hortonworks.com>
wrote:

> As per the docs, in Batch mode, dynamic memory allocation is avoided by
> storing messages being processed in ByteBuffers via Unsafe methods.
>
> Couldn't find any docs  describing mem mgmt in Streamingn mode. So...
>
> - Am wondering if this is also the case with Streaming ?
>
> - If so, how does Flink detect that an object is no longer being used and
> can be reclaimed for reuse once again ?
>
> -roshan
>

--001a11446af8cdf3f5053b5b0af3--