apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: Visitor API for DAG
Date Thu, 17 Nov 2016 17:06:44 GMT
Yes, It could happen after current DAG validation and before the
application master is launched.

- Tushar.


On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath <ram@datatorrent.com> wrote:
> When would the visits happen ? Just before normal validation ?
>
> Ram
>
> On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi <tushar@apache.org> wrote:
>
>> Hi All,
>>
>> How about adding visitor like API for DAG in Apex, and an api to
>> register visitor for the DAG.
>> Possible use cases are
>> -  Validator visitor which could validate the dag
>> -  Visitor to inject properties/attribute in the operator/streams from
>> some external sources.
>> -  Platform does not support validation of individual operators.
>> developer could write a validator visitor which would call validate
>> function of operator if it implements Validator interface.
>> - generate output schema based on operator config and input schema,
>> and set the schema on output stream.
>>
>> Sample API :
>>
>> dag.registerVisitor(DAGVisitor visitor);
>>
>> Call order of visitorFunctions.
>> - preVisitDAG(Attributes) // dag attributes
>>   for all operators
>>   - visitOperator(OperatorMeta meta) // access to operator, name,
>> attributes, properties
>>  ports
>>   - visitStream(StreamMeta meta) // access to
>> stream/name/attributes/properties/ports
>> - postVisitDAG()
>>
>> Regards,
>> -Tushar.
>>

Mime
View raw message