flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gyula Fora (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2497) Rework streaming iteration flow control
Date Fri, 07 Aug 2015 14:25:46 GMT
Gyula Fora created FLINK-2497:

             Summary: Rework streaming iteration flow control
                 Key: FLINK-2497
                 URL: https://issues.apache.org/jira/browse/FLINK-2497
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
            Reporter: Gyula Fora

Currently streaming iterations use a simple Blocking queue to pass data from the iteration
tails to the iteration heads.

While this works most of the time, during increased load (or even short bursts) the increased
amount of records in transit inside the loop can cause the limited number of input/output
buffer-pools to fill up and deadlock the system. (caused by the back-pressure mechanism)

One solution to this problem could be to remove back-pressure on the loop-edge by replacing
the Blocking queue with some soft spilling queue that will write the records automatically
to disk letting the iteration tail to progress with its input.

For a more robust solution we need to implement more elaborate flow control in streaming programs
containing iterations.

This message was sent by Atlassian JIRA

View raw message