edgent-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dale LaBossiere (JIRA)" <j...@apache.org>
Subject [jira] [Created] (QUARKS-211) Topology.events() uses unbounded PlumbingStreams.isolate(): can cause out of memory
Date Thu, 23 Jun 2016 21:09:16 GMT
Dale LaBossiere created QUARKS-211:

             Summary: Topology.events() uses unbounded PlumbingStreams.isolate(): can cause
out of memory
                 Key: QUARKS-211
                 URL: https://issues.apache.org/jira/browse/QUARKS-211
             Project: Quarks
          Issue Type: Bug
          Components: Runtime
            Reporter: Dale LaBossiere
            Priority: Critical

Using an unbounded isolate is problematic.  If the downstream processing can't keep up for
an extended period, ultimately the application can accumulate tuples until an "out of memory"
condition occurs.

Note: the javadoc says, see ``PlumbingStreams#pressureReliever()`` but the code is actually
using an **unbounded** ``PlumbingStreams#isolate()``

While an app could add its own bounded isolator to an events() generated stream, that adds
undesired per-tuple processing latency due to the additional isolation point.

Changing to a bounded scheme for the default seems better.  Either:
a) a bounded isolate (there's now an impl for that) and live with blocking the event listener
when the queue is full
b) a pressureReliever and live with dropping events/tuples when the queue is full.
Pick some not unreasonable bound list (e.g., 1000?) and document it.
I think "a" may be the lesser evil.

Additionally allow (require?) the Topology.events(eventSetup) handler to provide an "isolator

    ``events(Function<Consumer<T>,UnaryOperator<TStream<T>> eventSetup)``

Then eventSetup can:
    return null;  // use the (new) default isolator
or    return stream -> PlumbingStreams.pressureReducer(stream, queueSize);
or    stream -> PlumbingStreams.isolate(stream, queueSize);
or    Functions.identity() (or ``t -> t``) - for no isolation (maybe other characteristics
about the app eliminate the need/desire for this per-tuple isolation latency)
or ...  e.g., a "time bounded" pressureReducer or isolator, or a time&count bounded version
when such things exist

Looking for some consensus / votes for this.  I don't think we want our first release to have
the current behavior.

This message was sent by Atlassian JIRA

View raw message