flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-1326) The output serializers may consume excessive memory
Date Mon, 15 Dec 2014 11:44:13 GMT
Stephan Ewen created FLINK-1326:

             Summary: The output serializers may consume excessive memory
                 Key: FLINK-1326
                 URL: https://issues.apache.org/jira/browse/FLINK-1326
             Project: Flink
          Issue Type: Bug
          Components: Distributed Runtime
    Affects Versions: 0.8-incubating
            Reporter: Stephan Ewen
            Assignee: Stephan Ewen

The SpanningRecordSerializer creates large byte arrays to buffer records during serialization.
Since they never release the records, they may occupy a large amount of memory. Currently,
we use one per output partition, which is due to the original architecture.

There are two ways to fix this:

 1. Use only one serializer per output, rather than one per output partition. This is the
preferable way in the long run anyways.

 2. Release the arrays in the serializer (quick fix).

This message was sent by Atlassian JIRA

View raw message