apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandesh Hegde <sand...@datatorrent.com>
Subject Re: [Proposal] DAG listener
Date Tue, 16 Aug 2016 03:51:45 GMT
@Sanjay
Here is the issue I mentioned
https://issues.apache.org/jira/browse/APEXCORE-496

Instead of exposing one information at a time, why not publish all the
information?
To start with it could just be Physical DAG and update the subscribers
after partitioning.

Thanks


On Wed, Aug 10, 2016 at 10:40 AM Sanjay Pujare <sanjay@datatorrent.com>
wrote:

> Looks like a good idea but would like more details about the use case
> (“example ... to access the operator name while using Batched
> StatsListener”).
>
> Do you propose very fine grained subscription model? Also what “major
> activities in the DAG” will be available thru this listener?
>
>
>
> On 8/10/16, 8:20 AM, "Sandesh Hegde" <sandesh@datatorrent.com> wrote:
>
>     Any operators can subscribe to Stram Events affecting the DAG.
>
>     Implementation will most probably use heartbeat.
>
>     On Tue, Aug 9, 2016 at 11:43 PM Tushar Gosavi <tushar@datatorrent.com>
>     wrote:
>
>     > Hi Sandesh,
>     >
>     > Dag changes are handled in Stram, and operators are running in
>     > different containers.
>     > Are you suggesting an RPC interface or operator request for sending
>     > this information
>     > from Stram to all partitions of the interested operator?
>     >
>     > - Tushar.
>     >
>     >
>     >
>     > On Wed, Aug 10, 2016 at 11:28 AM, Sandesh Hegde <
> sandesh@datatorrent.com>
>     > wrote:
>     > > Hi All,
>     > >
>     > > As we add more features to support batch use cases, there will be
> a need
>     > to
>     > > access more information about the DAG from an operator. One
> example is
>     > the need
>     > > to access the operator name while using Batched StatsListener.
>     > >
>     > > The idea here is to implement DAG Listener ( similar to
> StatsListener ),
>     > > which can potentially give every information present in the Stram.
>     > > Operators will also have visibility into all the major activities
> in the
>     > > DAG.
>     > >
>     > > Thoughts?
>     > >
>     > > Thanks
>     >
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message