apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: Visitor API for DAG
Date Mon, 21 Nov 2016 06:43:20 GMT
Hi Ananth,

We can not change runtime properties through this API. The current
flow of apex application execution is

1) StramClient prepares application, inject properties from
properties.xml / user provided xml files and validates dag
2) StramClient copies required jars and serialized plan to HDFS and
launch master container.
3) Application master reads serialised plan from HDFS and starts
deploying StramClient as per deployment plan.

The visitor will examine the DAG in stage 1, hence visitor can only
change the initial state. The execution of the application is not
affected.

- Tushar.


On Sat, Nov 19, 2016 at 3:40 AM, ananth <ananthg.apex@gmail.com> wrote:
> How does this work for the stateful operators ? Can we use this to override
> properties that are deserialized ?
>
> Regards,
>
> Ananth
>
>
>
> On 18/11/16 05:53, Tushar Gosavi wrote:
>>
>> The code will execute before application master is launched, it is
>> just one time activity during application startup. Few use cases I
>> could think are
>>
>> - Operator validation/configuration validator
>>    jdbc operator could check if database is accessible with given
>> credentials.
>>    file output operator could if directory exists and filesystem is
>> writable.
>>
>> - Injection of properties in operators from external sources.
>>
>> - If two operator wants to exchange some information based on
>> configuration, they could do it through visitor. for example
>> TUPLE_SCHEMA can be set on downstream operator port based on operators
>> input TUPLE_SCHEMA and its configuration (for example projection
>> operator which drops few columns, could create a new class with fewer
>> fields and set it as tuple schema on downstream operator port).
>>
>> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a
>> efficient stream codec could be written using asm library for
>> serialisation and use that as stream codec instead of default one.
>>
>> -Tushar.
>>
>>
>> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <sanjay@datatorrent.com>
>> wrote:
>>>
>>> There is a risk if the user written code blocks the thread or crashes the
>>> process. What are the real life examples of this use case?
>>>
>>>
>>> On 11/17/16, 9:21 AM, "amol kekre" <amolhkekre@gmail.com> wrote:
>>>
>>>      +1. Opening up the API for users to put in their own code is good.
>>> In
>>>      general we should enable users to register their code in a lot of
>>> scenerios.
>>>
>>>      Thks
>>>      Amol
>>>
>>>      On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi
>>> <tushar@datatorrent.com>
>>>      wrote:
>>>
>>>      > Yes, It could happen after current DAG validation and before the
>>>      > application master is launched.
>>>      >
>>>      > - Tushar.
>>>      >
>>>      >
>>>      > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath
>>> <ram@datatorrent.com>
>>>      > wrote:
>>>      > > When would the visits happen ? Just before normal validation ?
>>>      > >
>>>      > > Ram
>>>      > >
>>>      > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi
>>> <tushar@apache.org>
>>>      > wrote:
>>>      > >
>>>      > >> Hi All,
>>>      > >>
>>>      > >> How about adding visitor like API for DAG in Apex, and an
api
>>> to
>>>      > >> register visitor for the DAG.
>>>      > >> Possible use cases are
>>>      > >> -  Validator visitor which could validate the dag
>>>      > >> -  Visitor to inject properties/attribute in the
>>> operator/streams from
>>>      > >> some external sources.
>>>      > >> -  Platform does not support validation of individual
>>> operators.
>>>      > >> developer could write a validator visitor which would call
>>> validate
>>>      > >> function of operator if it implements Validator interface.
>>>      > >> - generate output schema based on operator config and input
>>> schema,
>>>      > >> and set the schema on output stream.
>>>      > >>
>>>      > >> Sample API :
>>>      > >>
>>>      > >> dag.registerVisitor(DAGVisitor visitor);
>>>      > >>
>>>      > >> Call order of visitorFunctions.
>>>      > >> - preVisitDAG(Attributes) // dag attributes
>>>      > >>   for all operators
>>>      > >>   - visitOperator(OperatorMeta meta) // access to operator,
>>> name,
>>>      > >> attributes, properties
>>>      > >>  ports
>>>      > >>   - visitStream(StreamMeta meta) // access to
>>>      > >> stream/name/attributes/properties/ports
>>>      > >> - postVisitDAG()
>>>      > >>
>>>      > >> Regards,
>>>      > >> -Tushar.
>>>      > >>
>>>      >
>>>
>>>
>>>
>

Mime
View raw message