Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
From: Vlad Rozov <v.rozov@datatorrent.com>
Content-Type: text/plain;
	charset=koi8-r
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Date: Sat, 1 Apr 2017 12:04:22 -0700
Subject: Re: open/close ports and active/inactive streams
Message-Id: <D92DA000-67D6-4015-9647-258EA891B701@datatorrent.com>
References: <bee03d65-457a-41af-6909-5ff463be242b@datatorrent.com> <CAJ484BjzXzqQBEn9BbxX9FzUhLeM8r98FZSYmckemAw7zXuKKw@mail.gmail.com>
In-Reply-To: <CAJ484BjzXzqQBEn9BbxX9FzUhLeM8r98FZSYmckemAw7zXuKKw@mail.gmail.com>
To: dev@apex.apache.org
archived-at: Sat, 01 Apr 2017 19:04:31 -0000

One of use cases is a batch application with multiple reduce/aggregate opera=
tors.

Thank you,
Vlad

=EF=D4=D0=D2=C1=D7=CC=C5=CE=CF =D3 iPhone

> On Apr 1, 2017, at 10:23, Munagala Ramanath <ram@datatorrent.com> wrote:
>=20
> What's a use case for this ?
>=20
> Ram
>=20
>> On Sat, Apr 1, 2017 at 8:12 AM, Vlad Rozov <v.rozov@datatorrent.com> wrot=
e:
>>=20
>> All,
>>=20
>> Currently Apex assumes that an operator can emit on any defined output
>> port and all streams defined by a DAG are active. I'd like to propose an
>> ability for an operator to open and close output ports. By default all
>> ports defined by an operator will be open. In the case an operator for an=
y
>> reason decides that it will not emit tuples on the output port, it may
>> close it. This will make the stream inactive and the application master m=
ay
>> undeploy the downstream (for that input stream) operators. If this leads t=
o
>> containers that don't have any active operators, those containers may be
>> undeployed as well leading to better cluster resource utilization and
>> better Apex elasticity. Later, the operator may be in a state where it
>> needs to emit tuples on the closed port. In this case, it needs to re-ope=
n
>> the port and wait till the stream becomes active again before emitting
>> tuples on that port. Making inactive stream active again, requires the
>> application master to re-allocate containers and re-deploy the downstream=

>> operators.
>>=20
>> It should be also possible for an application designer to mark streams as=

>> inactive when an application starts. This will allow the application mast=
er
>> avoid reserving all containers when the application starts. Later, the po=
rt
>> can be open and inactive stream become active.
>>=20
>> Thank you,
>>=20
>> Vlad
>>=20
>>=20
>=20
>=20
> --=20
>=20
> _______________________________________________________
>=20
> Munagala V. Ramanath
>=20
> Software Engineer
>=20
> E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
>=20
> www.datatorrent.com  |  apex.apache.org