apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: open/close ports and active/inactive streams
Date Sat, 01 Apr 2017 17:23:58 GMT
What's a use case for this ?


On Sat, Apr 1, 2017 at 8:12 AM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> All,
> Currently Apex assumes that an operator can emit on any defined output
> port and all streams defined by a DAG are active. I'd like to propose an
> ability for an operator to open and close output ports. By default all
> ports defined by an operator will be open. In the case an operator for any
> reason decides that it will not emit tuples on the output port, it may
> close it. This will make the stream inactive and the application master may
> undeploy the downstream (for that input stream) operators. If this leads to
> containers that don't have any active operators, those containers may be
> undeployed as well leading to better cluster resource utilization and
> better Apex elasticity. Later, the operator may be in a state where it
> needs to emit tuples on the closed port. In this case, it needs to re-open
> the port and wait till the stream becomes active again before emitting
> tuples on that port. Making inactive stream active again, requires the
> application master to re-allocate containers and re-deploy the downstream
> operators.
> It should be also possible for an application designer to mark streams as
> inactive when an application starts. This will allow the application master
> avoid reserving all containers when the application starts. Later, the port
> can be open and inactive stream become active.
> Thank you,
> Vlad



Munagala V. Ramanath

Software Engineer

E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message