Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
From: Nico Kruber <nico@data-artisans.com>
To: yunfan123 <yunfanfighting@foxmail.com>
Cc: user@flink.apache.org, Chesnay Schepler <chesnay@apache.org>
Subject: Re: Flink takes too much memory in record serializer.
Date: Tue, 14 Nov 2017 14:25:57 +0100
Message-ID: <3266813.ngzCfZ3qxZ@nico-work>
Organization: data Artisans
In-Reply-To: <448691c1-39ab-a406-b1bb-a39cf954ac2a@apache.org>
References: <1510654284121-0.post@n4.nabble.com> <448691c1-39ab-a406-b1bb-a39cf954ac2a@apache.org>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="nextPart2357098.4g5vQYq2T5"; micalg="pgp-sha1"; protocol="application/pgp-signature"
archived-at: Tue, 14 Nov 2017 13:26:08 -0000

--nextPart2357098.4g5vQYq2T5
Content-Transfer-Encoding: 7Bit
Content-Type: text/plain; charset="us-ascii"

We're actually also trying to have the serializer stateless in future and may 
be able to remove the intermediate serialization buffer which is currently 
growing on heap before we copy the data into the actual target buffer. This 
intermediate buffer grows and is pruned after serialization if it is bigger 
than 5MB (see DataOutputSerializer), and re-used for anything below that 
threshold. So you may actually have up to 5MB per output channel which sits 
waiting for data.
Please refer to https://issues.apache.org/jira/browse/FLINK-4893 for updates 
on this.

These improvements will certainly reduce some of our memory footprint and help 
you. Throughput will then, of course, be limited by your network's speed and 
the number of network buffers to hold this amount of data and to saturate your 
network connections. The availability of these buffers will then limit your 
throughput accordingly.


Nico

On Tuesday, 14 November 2017 11:29:33 CET Chesnay Schepler wrote:
> I don't there's anything you can do except reducing the parallelism or
> the size of your messages.
> 
> A separate serializer is used for each channel as the serializers are
> stateful; they are capable of writing records partially
> to a given MemorySegment to better utilize the allocated memory.
> 
> How many messages is each operator instance processing per second? I
> would imagine that at this scale
> your memory consumption goes through the roof anyway due to the message
> size.
> Even if every operator instance is only processing 10 records/s you're
> already looking at 10TB memory usage
> for in-flight data.
> 
> On 14.11.2017 11:11, yunfan123 wrote:
> > In the class org.apache.flink.runtime.io.network.api.writer.RecordWriter,
> > it has same number of serializers with the numChannels.
> > If I first operator has 500 parallels and the next operator has 1000
> > parallels.
> > And every message in flink is 2MB.
> > The job takes 500 * 1000 * 2MB as 1TB memory in totally!!!
> > Can I do anything to reduce the memory usage.
> > 
> > 
> > 
> > --
> > Sent from:
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/


--nextPart2357098.4g5vQYq2T5
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part.
Content-Transfer-Encoding: 7Bit

-----BEGIN PGP SIGNATURE-----

iF0EABECAB0WIQTIh4KsbsNd3l7wd+cg8nJL2uqeWQUCWgru5QAKCRAg8nJL2uqe
WY8tAKCNXCNiZBS0Iaof4rxlFWc4fsbabwCgu8FALZt6XPwqu7Lzcn+2XH+vrwE=
=iFrv
-----END PGP SIGNATURE-----

--nextPart2357098.4g5vQYq2T5--