apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pramod Immaneni (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow
Date Sun, 15 Jan 2017 16:48:26 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823182#comment-15823182

Pramod Immaneni commented on APEXCORE-570:

I am not sure what is left to conclude the discussion but will add this. There are two scenarios
here when buffer spooling is enabled and when it is disabled. This addresses the case when
buffer spooling is enabled (the default case). The buffer spooling disabled case, has some
other basic issues, even before coming to back pressure, therefore requiring a different treatment
and I am planning to look at that next, the issues here are captured in a separate jira APEXCORE-609.

Let me describe the approach by first describing what happens today. The data in the buffer
server is stored in blocks. For example, if you have a 512MB buffer (the default) it is divided
into 8 blocks of 64MB each. The problem is that the as the publisher publishes more data,
the blocks it is done with are spooled to disk, their data memory invalidated, regardless
of where the subscribers are, and new blocks are allocated with total data memory usage remaining
at 512MB (configured capacity). This is done immaterial of where the subscribers are in terms
of reading those blocks. If a subscriber hits a block that is spooled that data is loaded
back into memory. My approach is simply to not release blocks haphazardly when publisher is
done with a block but do that only after all subscribers have read them and if there are no
more blocks left for a publisher, then suspend publisher till the slowest subscriber has gone
past the earliest block and released it.

If you now have an app where the input operator can bring data into the dag at a much faster
rate than lets say an output operator can write to a store, you will see the individual buffers
in the intermediate containers build up and those operators pause as they reach the limit,
with the back pressure propagating all the way back to the input operator and pausing it eventually,
and as the output operator moves forward the upstream operators resume. So the buffer memory
limit specified by the configurable attribute controls when the back pressure will kick in.
Of course this is not an ideal way to design/run your app and you want an impedance match
between your input and output or at least avg(input rate) <= avg(output rate), but in scenarios
where the application is not designed to do this, it prevents a runaway happening with the
upstream operators. By the way the feature can be disabled if not desired.

> Prevent upstream operators from getting too far ahead when downstream operators are slow
> ----------------------------------------------------------------------------------------
>                 Key: APEXCORE-570
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-570
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Pramod Immaneni
>            Assignee: Pramod Immaneni
> If the downstream operators are slower than upstream operators then the upstream operators
will get ahead and the gap can continue to increase. Provide an option to slow down or temporarily
pause the upstream operators when they get too far ahead.

This message was sent by Atlassian JIRA

View raw message