apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Pujare <san...@datatorrent.com>
Subject Re: open/close ports and active/inactive streams
Date Sat, 01 Apr 2017 16:13:35 GMT
Sounds like a good idea, +1

Couple of questions/comments:
- "...re-open the port and wait till the stream becomes active again...".
How is the operator informed that the stream is active again or is it just
a property of the output port that the operator needs to check?
- Say an operator has 2 input ports one of which becomes inactive as per
this scenario. It cannot be undeployed because of the other port but the
operator needs to behave differently because it's a different DAG now. Will
a control tuple inform it that the input port is closing?

On Sat, Apr 1, 2017 at 8:12 AM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> All,
> Currently Apex assumes that an operator can emit on any defined output
> port and all streams defined by a DAG are active. I'd like to propose an
> ability for an operator to open and close output ports. By default all
> ports defined by an operator will be open. In the case an operator for any
> reason decides that it will not emit tuples on the output port, it may
> close it. This will make the stream inactive and the application master may
> undeploy the downstream (for that input stream) operators. If this leads to
> containers that don't have any active operators, those containers may be
> undeployed as well leading to better cluster resource utilization and
> better Apex elasticity. Later, the operator may be in a state where it
> needs to emit tuples on the closed port. In this case, it needs to re-open
> the port and wait till the stream becomes active again before emitting
> tuples on that port. Making inactive stream active again, requires the
> application master to re-allocate containers and re-deploy the downstream
> operators.
> It should be also possible for an application designer to mark streams as
> inactive when an application starts. This will allow the application master
> avoid reserving all containers when the application starts. Later, the port
> can be open and inactive stream become active.
> Thank you,
> Vlad

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message