apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <v.ro...@datatorrent.com>
Subject Re: open/close ports and active/inactive streams
Date Sat, 01 Apr 2017 19:04:22 GMT
One of use cases is a batch application with multiple reduce/aggregate operators.

Thank you,
Vlad

Отправлено с iPhone

> On Apr 1, 2017, at 10:23, Munagala Ramanath <ram@datatorrent.com> wrote:
> 
> What's a use case for this ?
> 
> Ram
> 
>> On Sat, Apr 1, 2017 at 8:12 AM, Vlad Rozov <v.rozov@datatorrent.com> wrote:
>> 
>> All,
>> 
>> Currently Apex assumes that an operator can emit on any defined output
>> port and all streams defined by a DAG are active. I'd like to propose an
>> ability for an operator to open and close output ports. By default all
>> ports defined by an operator will be open. In the case an operator for any
>> reason decides that it will not emit tuples on the output port, it may
>> close it. This will make the stream inactive and the application master may
>> undeploy the downstream (for that input stream) operators. If this leads to
>> containers that don't have any active operators, those containers may be
>> undeployed as well leading to better cluster resource utilization and
>> better Apex elasticity. Later, the operator may be in a state where it
>> needs to emit tuples on the closed port. In this case, it needs to re-open
>> the port and wait till the stream becomes active again before emitting
>> tuples on that port. Making inactive stream active again, requires the
>> application master to re-allocate containers and re-deploy the downstream
>> operators.
>> 
>> It should be also possible for an application designer to mark streams as
>> inactive when an application starts. This will allow the application master
>> avoid reserving all containers when the application starts. Later, the port
>> can be open and inactive stream become active.
>> 
>> Thank you,
>> 
>> Vlad
>> 
>> 
> 
> 
> -- 
> 
> _______________________________________________________
> 
> Munagala V. Ramanath
> 
> Software Engineer
> 
> E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
> 
> www.datatorrent.com  |  apex.apache.org

Mime
View raw message