Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Thu, 1 Jun 2017 16:20:04 +0000 (UTC)
From: "Antony Mayi (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13076577.1496333903000.342820.1496334004320@Atlassian.JIRA>
In-Reply-To: <JIRA.13076577.1496333903000@Atlassian.JIRA>
References: <JIRA.13076577.1496333903000@Atlassian.JIRA> <JIRA.13076577.1496333903750@jira-lw-us.apache.org>
Subject: [jira] [Updated] (BEAM-2398) Increasing latency within DirectRunner
 caused by cumulated TransformWatermarks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 01 Jun 2017 16:20:16 -0000


     [ https://issues.apache.org/jira/browse/BEAM-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Antony Mayi updated BEAM-2398:
------------------------------
    Description: 
Over the time the end-to-end latency of a pipeline running on DirectRunner is significantly increasing.

This is caused by ever growing sets of:
* {{WatermarkManager.TransformWatermarks.inputWatermark.pendingElements}}
* {{WatermarkManager.TransformWatermarks.synchronizedProcessingInputWatermark.pendingBundles}}

That means calls to {{WatermarkManager.TransformWatermarks.refresh()}} which need to iterate through that collections take longer and longer and the latency is growing.

I believe it is the line {{WaterMark.updatePending()}} line:

{quote}
    if (input != null) {
      // Add the unprocessed inputs
      completedTransform.addPending(result.getUnprocessedInputs());
{quote}
that's adding the items that are never removed.

See attached [^LatencyTest.java] demo code showing the increasing latency.

  was:
Over the time the end-to-end latency of a pipeline running on DirectRunner is significantly increasing.

This is caused by ever growing sets of:
* {{WatermarkManager.TransformWatermarks.inputWatermark.pendingElements}}
* {{WatermarkManager.TransformWatermarks.synchronizedProcessingInputWatermark.pendingBundles}}

That means calls to {{WatermarkManager.TransformWatermarks.refresh()}} which need to iterate through that collections take longer and longer and the latency is growing.

I believe it is the line {{WaterMark.updatePending()}} line:

{quote}
    if (input != null) {
      // Add the unprocessed inputs
      completedTransform.addPending(result.getUnprocessedInputs());
{quote}
that's adding the items that are never removed.

See attached demo code showing the increasing latency.


> Increasing latency within DirectRunner caused by cumulated TransformWatermarks
> ------------------------------------------------------------------------------
>
>                 Key: BEAM-2398
>                 URL: https://issues.apache.org/jira/browse/BEAM-2398
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-direct
>    Affects Versions: 2.0.0
>            Reporter: Antony Mayi
>            Assignee: Thomas Groh
>         Attachments: LatencyTest.java
>
>
> Over the time the end-to-end latency of a pipeline running on DirectRunner is significantly increasing.
> This is caused by ever growing sets of:
> * {{WatermarkManager.TransformWatermarks.inputWatermark.pendingElements}}
> * {{WatermarkManager.TransformWatermarks.synchronizedProcessingInputWatermark.pendingBundles}}
> That means calls to {{WatermarkManager.TransformWatermarks.refresh()}} which need to iterate through that collections take longer and longer and the latency is growing.
> I believe it is the line {{WaterMark.updatePending()}} line:
> {quote}
>     if (input != null) {
>       // Add the unprocessed inputs
>       completedTransform.addPending(result.getUnprocessedInputs());
> {quote}
> that's adding the items that are never removed.
> See attached [^LatencyTest.java] demo code showing the increasing latency.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)