Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Fri, 7 Aug 2015 14:25:46 +0000 (UTC)
From: "Gyula Fora (JIRA)" <jira@apache.org>
To: dev@flink.apache.org
Message-ID: <JIRA.12852968.1438957539000.5578.1438957546574@Atlassian.JIRA>
In-Reply-To: <JIRA.12852968.1438957539000@Atlassian.JIRA>
References: <JIRA.12852968.1438957539000@Atlassian.JIRA>
 <JIRA.12852968.1438957539148@arcas>
Subject: [jira] [Created] (FLINK-2497) Rework streaming iteration flow
 control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Gyula Fora created FLINK-2497:
---------------------------------

             Summary: Rework streaming iteration flow control
                 Key: FLINK-2497
                 URL: https://issues.apache.org/jira/browse/FLINK-2497
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
            Reporter: Gyula Fora


Currently streaming iterations use a simple Blocking queue to pass data from the iteration tails to the iteration heads.

While this works most of the time, during increased load (or even short bursts) the increased amount of records in transit inside the loop can cause the limited number of input/output buffer-pools to fill up and deadlock the system. (caused by the back-pressure mechanism)

One solution to this problem could be to remove back-pressure on the loop-edge by replacing the Blocking queue with some soft spilling queue that will write the records automatically to disk letting the iteration tail to progress with its input.

For a more robust solution we need to implement more elaborate flow control in streaming programs containing iterations.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)