apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Weise (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow
Date Sun, 15 Jan 2017 19:16:26 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823228#comment-15823228

Thomas Weise commented on APEXCORE-570:

Let’s first address the process issue (it may warrant a separate discussion and additions
to the contributor guidelines also). If you think there was a conclusion then this may indicate
that there was offline discussion that isn’t captured here or anywhere else. Just by looking
at this ticket it is everything but clear what lead to your PR. This is not how the community
can work, discussion has to be in the open. 

Now back to the topic: I think this is a good approach. It will avoid the fast producing operator
running ahead at the speed of writing to disk indefinitely. With this we will not need to
limit the spooling at all?

What about the case where you have two subscribers (and those could be different operators)
where one can keep up with the rate at which data is published and the other one may be slow,
albeit maybe temporarily? This will slow down the fast subscriber and introduce latency.

> Prevent upstream operators from getting too far ahead when downstream operators are slow
> ----------------------------------------------------------------------------------------
>                 Key: APEXCORE-570
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-570
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Pramod Immaneni
>            Assignee: Pramod Immaneni
> If the downstream operators are slower than upstream operators then the upstream operators
will get ahead and the gap can continue to increase. Provide an option to slow down or temporarily
pause the upstream operators when they get too far ahead.

This message was sent by Atlassian JIRA

View raw message