apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: Visitor API for DAG
Date Fri, 25 Nov 2016 11:24:37 GMT
Opened a Jira https://issues.apache.org/jira/browse/APEXCORE-577 for this.

- Tushar.


On Mon, Nov 21, 2016 at 9:59 PM, Amol Kekre <amol@datatorrent.com> wrote:
> Ananth,
> The current API allows changing properties of a running app. The new
> proposed API is not needed to do so.
>
> Thks
> Amol
>
>
> On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
>
>> Hi Ananth,
>>
>> We can not change runtime properties through this API. The current
>> flow of apex application execution is
>>
>> 1) StramClient prepares application, inject properties from
>> properties.xml / user provided xml files and validates dag
>> 2) StramClient copies required jars and serialized plan to HDFS and
>> launch master container.
>> 3) Application master reads serialised plan from HDFS and starts
>> deploying StramClient as per deployment plan.
>>
>> The visitor will examine the DAG in stage 1, hence visitor can only
>> change the initial state. The execution of the application is not
>> affected.
>>
>> - Tushar.
>>
>>
>> On Sat, Nov 19, 2016 at 3:40 AM, ananth <ananthg.apex@gmail.com> wrote:
>> > How does this work for the stateful operators ? Can we use this to
>> override
>> > properties that are deserialized ?
>> >
>> > Regards,
>> >
>> > Ananth
>> >
>> >
>> >
>> > On 18/11/16 05:53, Tushar Gosavi wrote:
>> >>
>> >> The code will execute before application master is launched, it is
>> >> just one time activity during application startup. Few use cases I
>> >> could think are
>> >>
>> >> - Operator validation/configuration validator
>> >>    jdbc operator could check if database is accessible with given
>> >> credentials.
>> >>    file output operator could if directory exists and filesystem is
>> >> writable.
>> >>
>> >> - Injection of properties in operators from external sources.
>> >>
>> >> - If two operator wants to exchange some information based on
>> >> configuration, they could do it through visitor. for example
>> >> TUPLE_SCHEMA can be set on downstream operator port based on operators
>> >> input TUPLE_SCHEMA and its configuration (for example projection
>> >> operator which drops few columns, could create a new class with fewer
>> >> fields and set it as tuple schema on downstream operator port).
>> >>
>> >> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a
>> >> efficient stream codec could be written using asm library for
>> >> serialisation and use that as stream codec instead of default one.
>> >>
>> >> -Tushar.
>> >>
>> >>
>> >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <sanjay@datatorrent.com
>> >
>> >> wrote:
>> >>>
>> >>> There is a risk if the user written code blocks the thread or crashes
>> the
>> >>> process. What are the real life examples of this use case?
>> >>>
>> >>>
>> >>> On 11/17/16, 9:21 AM, "amol kekre" <amolhkekre@gmail.com> wrote:
>> >>>
>> >>>      +1. Opening up the API for users to put in their own code is good.
>> >>> In
>> >>>      general we should enable users to register their code in a lot
of
>> >>> scenerios.
>> >>>
>> >>>      Thks
>> >>>      Amol
>> >>>
>> >>>      On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi
>> >>> <tushar@datatorrent.com>
>> >>>      wrote:
>> >>>
>> >>>      > Yes, It could happen after current DAG validation and before
the
>> >>>      > application master is launched.
>> >>>      >
>> >>>      > - Tushar.
>> >>>      >
>> >>>      >
>> >>>      > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath
>> >>> <ram@datatorrent.com>
>> >>>      > wrote:
>> >>>      > > When would the visits happen ? Just before normal validation
?
>> >>>      > >
>> >>>      > > Ram
>> >>>      > >
>> >>>      > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi
>> >>> <tushar@apache.org>
>> >>>      > wrote:
>> >>>      > >
>> >>>      > >> Hi All,
>> >>>      > >>
>> >>>      > >> How about adding visitor like API for DAG in Apex,
and an api
>> >>> to
>> >>>      > >> register visitor for the DAG.
>> >>>      > >> Possible use cases are
>> >>>      > >> -  Validator visitor which could validate the dag
>> >>>      > >> -  Visitor to inject properties/attribute in the
>> >>> operator/streams from
>> >>>      > >> some external sources.
>> >>>      > >> -  Platform does not support validation of individual
>> >>> operators.
>> >>>      > >> developer could write a validator visitor which would
call
>> >>> validate
>> >>>      > >> function of operator if it implements Validator interface.
>> >>>      > >> - generate output schema based on operator config
and input
>> >>> schema,
>> >>>      > >> and set the schema on output stream.
>> >>>      > >>
>> >>>      > >> Sample API :
>> >>>      > >>
>> >>>      > >> dag.registerVisitor(DAGVisitor visitor);
>> >>>      > >>
>> >>>      > >> Call order of visitorFunctions.
>> >>>      > >> - preVisitDAG(Attributes) // dag attributes
>> >>>      > >>   for all operators
>> >>>      > >>   - visitOperator(OperatorMeta meta) // access to
operator,
>> >>> name,
>> >>>      > >> attributes, properties
>> >>>      > >>  ports
>> >>>      > >>   - visitStream(StreamMeta meta) // access to
>> >>>      > >> stream/name/attributes/properties/ports
>> >>>      > >> - postVisitDAG()
>> >>>      > >>
>> >>>      > >> Regards,
>> >>>      > >> -Tushar.
>> >>>      > >>
>> >>>      >
>> >>>
>> >>>
>> >>>
>> >
>>

Mime
View raw message