apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ananth <ananthg.a...@gmail.com>
Subject Re: Visitor API for DAG
Date Fri, 18 Nov 2016 22:10:44 GMT
How does this work for the stateful operators ? Can we use this to 
override properties that are deserialized ?

Regards,

Ananth


On 18/11/16 05:53, Tushar Gosavi wrote:
> The code will execute before application master is launched, it is
> just one time activity during application startup. Few use cases I
> could think are
>
> - Operator validation/configuration validator
>    jdbc operator could check if database is accessible with given credentials.
>    file output operator could if directory exists and filesystem is writable.
>
> - Injection of properties in operators from external sources.
>
> - If two operator wants to exchange some information based on
> configuration, they could do it through visitor. for example
> TUPLE_SCHEMA can be set on downstream operator port based on operators
> input TUPLE_SCHEMA and its configuration (for example projection
> operator which drops few columns, could create a new class with fewer
> fields and set it as tuple schema on downstream operator port).
>
> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a
> efficient stream codec could be written using asm library for
> serialisation and use that as stream codec instead of default one.
>
> -Tushar.
>
>
> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <sanjay@datatorrent.com> wrote:
>> There is a risk if the user written code blocks the thread or crashes the process.
What are the real life examples of this use case?
>>
>>
>> On 11/17/16, 9:21 AM, "amol kekre" <amolhkekre@gmail.com> wrote:
>>
>>      +1. Opening up the API for users to put in their own code is good. In
>>      general we should enable users to register their code in a lot of scenerios.
>>
>>      Thks
>>      Amol
>>
>>      On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi <tushar@datatorrent.com>
>>      wrote:
>>
>>      > Yes, It could happen after current DAG validation and before the
>>      > application master is launched.
>>      >
>>      > - Tushar.
>>      >
>>      >
>>      > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath <ram@datatorrent.com>
>>      > wrote:
>>      > > When would the visits happen ? Just before normal validation ?
>>      > >
>>      > > Ram
>>      > >
>>      > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi <tushar@apache.org>
>>      > wrote:
>>      > >
>>      > >> Hi All,
>>      > >>
>>      > >> How about adding visitor like API for DAG in Apex, and an api
to
>>      > >> register visitor for the DAG.
>>      > >> Possible use cases are
>>      > >> -  Validator visitor which could validate the dag
>>      > >> -  Visitor to inject properties/attribute in the operator/streams
from
>>      > >> some external sources.
>>      > >> -  Platform does not support validation of individual operators.
>>      > >> developer could write a validator visitor which would call validate
>>      > >> function of operator if it implements Validator interface.
>>      > >> - generate output schema based on operator config and input schema,
>>      > >> and set the schema on output stream.
>>      > >>
>>      > >> Sample API :
>>      > >>
>>      > >> dag.registerVisitor(DAGVisitor visitor);
>>      > >>
>>      > >> Call order of visitorFunctions.
>>      > >> - preVisitDAG(Attributes) // dag attributes
>>      > >>   for all operators
>>      > >>   - visitOperator(OperatorMeta meta) // access to operator, name,
>>      > >> attributes, properties
>>      > >>  ports
>>      > >>   - visitStream(StreamMeta meta) // access to
>>      > >> stream/name/attributes/properties/ports
>>      > >> - postVisitDAG()
>>      > >>
>>      > >> Regards,
>>      > >> -Tushar.
>>      > >>
>>      >
>>
>>
>>


Mime
View raw message