apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Visitor API for DAG
Date Mon, 21 Nov 2016 16:29:15 GMT
Ananth,
The current API allows changing properties of a running app. The new
proposed API is not needed to do so.

Thks
Amol


On Sun, Nov 20, 2016 at 10:43 PM, Tushar Gosavi <tushar@datatorrent.com>
wrote:

> Hi Ananth,
>
> We can not change runtime properties through this API. The current
> flow of apex application execution is
>
> 1) StramClient prepares application, inject properties from
> properties.xml / user provided xml files and validates dag
> 2) StramClient copies required jars and serialized plan to HDFS and
> launch master container.
> 3) Application master reads serialised plan from HDFS and starts
> deploying StramClient as per deployment plan.
>
> The visitor will examine the DAG in stage 1, hence visitor can only
> change the initial state. The execution of the application is not
> affected.
>
> - Tushar.
>
>
> On Sat, Nov 19, 2016 at 3:40 AM, ananth <ananthg.apex@gmail.com> wrote:
> > How does this work for the stateful operators ? Can we use this to
> override
> > properties that are deserialized ?
> >
> > Regards,
> >
> > Ananth
> >
> >
> >
> > On 18/11/16 05:53, Tushar Gosavi wrote:
> >>
> >> The code will execute before application master is launched, it is
> >> just one time activity during application startup. Few use cases I
> >> could think are
> >>
> >> - Operator validation/configuration validator
> >>    jdbc operator could check if database is accessible with given
> >> credentials.
> >>    file output operator could if directory exists and filesystem is
> >> writable.
> >>
> >> - Injection of properties in operators from external sources.
> >>
> >> - If two operator wants to exchange some information based on
> >> configuration, they could do it through visitor. for example
> >> TUPLE_SCHEMA can be set on downstream operator port based on operators
> >> input TUPLE_SCHEMA and its configuration (for example projection
> >> operator which drops few columns, could create a new class with fewer
> >> fields and set it as tuple schema on downstream operator port).
> >>
> >> - For pojo enabled operator (port where TUPLE_SCHEMA is defined), a
> >> efficient stream codec could be written using asm library for
> >> serialisation and use that as stream codec instead of default one.
> >>
> >> -Tushar.
> >>
> >>
> >> On Thu, Nov 17, 2016 at 11:35 PM, Sanjay Pujare <sanjay@datatorrent.com
> >
> >> wrote:
> >>>
> >>> There is a risk if the user written code blocks the thread or crashes
> the
> >>> process. What are the real life examples of this use case?
> >>>
> >>>
> >>> On 11/17/16, 9:21 AM, "amol kekre" <amolhkekre@gmail.com> wrote:
> >>>
> >>>      +1. Opening up the API for users to put in their own code is good.
> >>> In
> >>>      general we should enable users to register their code in a lot of
> >>> scenerios.
> >>>
> >>>      Thks
> >>>      Amol
> >>>
> >>>      On Thu, Nov 17, 2016 at 9:06 AM, Tushar Gosavi
> >>> <tushar@datatorrent.com>
> >>>      wrote:
> >>>
> >>>      > Yes, It could happen after current DAG validation and before the
> >>>      > application master is launched.
> >>>      >
> >>>      > - Tushar.
> >>>      >
> >>>      >
> >>>      > On Thu, Nov 17, 2016 at 8:32 PM, Munagala Ramanath
> >>> <ram@datatorrent.com>
> >>>      > wrote:
> >>>      > > When would the visits happen ? Just before normal validation
?
> >>>      > >
> >>>      > > Ram
> >>>      > >
> >>>      > > On Wed, Nov 16, 2016 at 9:50 PM, Tushar Gosavi
> >>> <tushar@apache.org>
> >>>      > wrote:
> >>>      > >
> >>>      > >> Hi All,
> >>>      > >>
> >>>      > >> How about adding visitor like API for DAG in Apex, and
an api
> >>> to
> >>>      > >> register visitor for the DAG.
> >>>      > >> Possible use cases are
> >>>      > >> -  Validator visitor which could validate the dag
> >>>      > >> -  Visitor to inject properties/attribute in the
> >>> operator/streams from
> >>>      > >> some external sources.
> >>>      > >> -  Platform does not support validation of individual
> >>> operators.
> >>>      > >> developer could write a validator visitor which would
call
> >>> validate
> >>>      > >> function of operator if it implements Validator interface.
> >>>      > >> - generate output schema based on operator config and
input
> >>> schema,
> >>>      > >> and set the schema on output stream.
> >>>      > >>
> >>>      > >> Sample API :
> >>>      > >>
> >>>      > >> dag.registerVisitor(DAGVisitor visitor);
> >>>      > >>
> >>>      > >> Call order of visitorFunctions.
> >>>      > >> - preVisitDAG(Attributes) // dag attributes
> >>>      > >>   for all operators
> >>>      > >>   - visitOperator(OperatorMeta meta) // access to operator,
> >>> name,
> >>>      > >> attributes, properties
> >>>      > >>  ports
> >>>      > >>   - visitStream(StreamMeta meta) // access to
> >>>      > >> stream/name/attributes/properties/ports
> >>>      > >> - postVisitDAG()
> >>>      > >>
> >>>      > >> Regards,
> >>>      > >> -Tushar.
> >>>      > >>
> >>>      >
> >>>
> >>>
> >>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message