apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: [APEX-107] Support for specifying module properties on modules.
Date Mon, 30 Nov 2015 16:41:31 GMT
addOperator method in DAG interface takes object of type Operator, unless
we make Module as subclass of Operator we can not use the same API.

- Tushar.


On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <thomas@datatorrent.com>
wrote:

> +1
>
> If there are really module specific constructs, they can be added later.
>
> Why is the same not supported in the Java API, i.e. why not allow a module
> to be added like an operator and handle the distinction in the
> implementation?
>
> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > I am going to add a feature to allow use of modules in property file and
> > json api.
> > I am planing to use same "operators" array for adding modules as well as
> > operators. The reasoning behind is that user don't have to worry about
> the
> > component being added to the DAG is actually an operator or a module, as
> > the method to specify their properties and to create instance of them is
> > same. And it results in less complex syntax for writing application using
> > json/property file.
> >
> > For example the sample Application specified using Json will look like
> > below.
> > ```json
> > {
> >   "operators": [
> >     {
> >       "name": "operator1",
> >       "class": "com.datatorrent.lib.operator.Input",
> >       "properties": {
> >         "property1": "value1"
> >       }
> >     },
> >     {
> >       "name": "module1",
> >       "class": "com.datatorrent.module.Module1",
> >       "properties": {
> >         "property1": "value1"
> >       }
> >     },
> >   ],
> >   "streams": [
> >     {
> >       "name": "s1",
> >       "source": {
> >         "operatorName": "operator1",
> >         "portName": "output"
> >       },
> >       "sinks": [
> >         {
> >           "operatorName": "module",
> >           "portName": "input"
> >         }
> >       ]
> >     }
> >   ]
> > }
> > ```
> >
> > In above example "module1" is of type Module. While processing the Json
> if
> > the
> > class of component is of type Module then we will use addModule api to
> the
> > DAG.
> >
> > Let me know, what do you think about this.
> >
> > Regards,
> > - Tushar.
> >
> > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tushar@datatorrent.com>
> > wrote:
> >
> > >
> > > A module is superset of operator before expansion in logical plan,
> except
> > > operator
> > > attribute are not applicable to modules. As Tim suggested we will have
> to
> > > add checks
> > > later to validate these attributes settings, and not allow top level
> > > operator and
> > > module to share same name.
> > >
> > > The same question comes when we want to add module to application using
> > > property
> > > file API and JSON file API.
> > >
> > > dt.operator.<operatorname>.classname=<fqcn of operator>
> > >
> > > One approach we can take is, if class is of type module, then use
> > > addModule api
> > > to add module to the dag, else use addOperator api. The initialization
> of
> > > object and applying property is generic can be reused by module.
> > >
> > > Similarly we have Json API where all operators are specified under
> > > operators array
> > >
> > > {
> > >   "operators": [
> > >     {.. operators .. }
> > >   ],
> > >   "streams": [
> > >     {.. streams .. }
> > >   ]
> > > }
> > >
> > > We could add module in the operators array, and based on the object
> type
> > > we can
> > > consider it as module or operator. The mechanism of initializing object
> > > and applying
> > > properties remains same.
> > >
> > > - Tushar.
> > >
> > >
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message