flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Flink takes too much memory in record serializer.
Date Tue, 14 Nov 2017 10:29:33 GMT
I don't there's anything you can do except reducing the parallelism or 
the size of your messages.

A separate serializer is used for each channel as the serializers are 
stateful; they are capable of writing records partially
to a given MemorySegment to better utilize the allocated memory.

How many messages is each operator instance processing per second? I 
would imagine that at this scale
your memory consumption goes through the roof anyway due to the message 
size.
Even if every operator instance is only processing 10 records/s you're 
already looking at 10TB memory usage
for in-flight data.

On 14.11.2017 11:11, yunfan123 wrote:
> In the class org.apache.flink.runtime.io.network.api.writer.RecordWriter, it
> has same number of serializers with the numChannels.
> If I first operator has 500 parallels and the next operator has 1000
> parallels.
> And every message in flink is 2MB.
> The job takes 500 * 1000 * 2MB as 1TB memory in totally!!!
> Can I do anything to reduce the memory usage.
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>


Mime
View raw message