apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: [APEX-107] Support for specifying module properties on modules.
Date Tue, 01 Dec 2015 05:46:52 GMT
+1. Modules and operators should be interchangeable.

Should something be recommend in terms of naming convention for Module
Class in that case?

- Chinmay.

~ Chinmay.

On Tue, Dec 1, 2015 at 9:51 AM, Tushar Gosavi <tushar@datatorrent.com>
wrote:

> Good point Vlad, if the class implements both Operator and Module
> interface. We will consider it as a module by default in json API, But also
> provide a way to specify his intention whether it should be considered as
> Module or Operator.
>
> If user use a Java API then intention is clear whether the object should be
> considered as Module or Operator (addOperator v/s addModule), will it be
> correct to change the intention of the user internally based on the type of
> component?
>
> - Tushar.
>
>
> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v.rozov@datatorrent.com>
> wrote:
>
> > +1. Modules and operators should be interchangeable. It should be
> possible
> > for an operator designer to replace it with a module without requirement
> to
> > recompile all applications that use the existing operator. From
> application
> > designer view both operators and modules may be considered as a black box
> > with a private implementation that may be monolithic (operator) or
> > distributed (module).
> >
> > Note that a Java class may implement both Operator and Module interface
> > and be added to a DAG using addOperator() method. At run-time (Logical
> > Plan->Physical Plan) Apex platform should handle such classes as modules
> > and may not depend on how a module was added to a DAG, IMO.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 11/30/15 08:41, Tushar Gosavi wrote:
> >
> >> addOperator method in DAG interface takes object of type Operator,
> unless
> >> we make Module as subclass of Operator we can not use the same API.
> >>
> >> - Tushar.
> >>
> >>
> >> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <thomas@datatorrent.com>
> >> wrote:
> >>
> >> +1
> >>>
> >>> If there are really module specific constructs, they can be added
> later.
> >>>
> >>> Why is the same not supported in the Java API, i.e. why not allow a
> >>> module
> >>> to be added like an operator and handle the distinction in the
> >>> implementation?
> >>>
> >>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
> >
> >>> wrote:
> >>>
> >>> Hi All,
> >>>>
> >>>> I am going to add a feature to allow use of modules in property file
> and
> >>>> json api.
> >>>> I am planing to use same "operators" array for adding modules as well
> as
> >>>> operators. The reasoning behind is that user don't have to worry about
> >>>>
> >>> the
> >>>
> >>>> component being added to the DAG is actually an operator or a module,
> as
> >>>> the method to specify their properties and to create instance of them
> is
> >>>> same. And it results in less complex syntax for writing application
> >>>> using
> >>>> json/property file.
> >>>>
> >>>> For example the sample Application specified using Json will look like
> >>>> below.
> >>>> ```json
> >>>> {
> >>>>    "operators": [
> >>>>      {
> >>>>        "name": "operator1",
> >>>>        "class": "com.datatorrent.lib.operator.Input",
> >>>>        "properties": {
> >>>>          "property1": "value1"
> >>>>        }
> >>>>      },
> >>>>      {
> >>>>        "name": "module1",
> >>>>        "class": "com.datatorrent.module.Module1",
> >>>>        "properties": {
> >>>>          "property1": "value1"
> >>>>        }
> >>>>      },
> >>>>    ],
> >>>>    "streams": [
> >>>>      {
> >>>>        "name": "s1",
> >>>>        "source": {
> >>>>          "operatorName": "operator1",
> >>>>          "portName": "output"
> >>>>        },
> >>>>        "sinks": [
> >>>>          {
> >>>>            "operatorName": "module",
> >>>>            "portName": "input"
> >>>>          }
> >>>>        ]
> >>>>      }
> >>>>    ]
> >>>> }
> >>>> ```
> >>>>
> >>>> In above example "module1" is of type Module. While processing the
> Json
> >>>>
> >>> if
> >>>
> >>>> the
> >>>> class of component is of type Module then we will use addModule api
to
> >>>>
> >>> the
> >>>
> >>>> DAG.
> >>>>
> >>>> Let me know, what do you think about this.
> >>>>
> >>>> Regards,
> >>>> - Tushar.
> >>>>
> >>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
> tushar@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>> A module is superset of operator before expansion in logical plan,
> >>>>>
> >>>> except
> >>>
> >>>> operator
> >>>>> attribute are not applicable to modules. As Tim suggested we will
> have
> >>>>>
> >>>> to
> >>>
> >>>> add checks
> >>>>> later to validate these attributes settings, and not allow top level
> >>>>> operator and
> >>>>> module to share same name.
> >>>>>
> >>>>> The same question comes when we want to add module to application
> using
> >>>>> property
> >>>>> file API and JSON file API.
> >>>>>
> >>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
> >>>>>
> >>>>> One approach we can take is, if class is of type module, then use
> >>>>> addModule api
> >>>>> to add module to the dag, else use addOperator api. The
> initialization
> >>>>>
> >>>> of
> >>>
> >>>> object and applying property is generic can be reused by module.
> >>>>>
> >>>>> Similarly we have Json API where all operators are specified under
> >>>>> operators array
> >>>>>
> >>>>> {
> >>>>>    "operators": [
> >>>>>      {.. operators .. }
> >>>>>    ],
> >>>>>    "streams": [
> >>>>>      {.. streams .. }
> >>>>>    ]
> >>>>> }
> >>>>>
> >>>>> We could add module in the operators array, and based on the object
> >>>>>
> >>>> type
> >>>
> >>>> we can
> >>>>> consider it as module or operator. The mechanism of initializing
> object
> >>>>> and applying
> >>>>> properties remains same.
> >>>>>
> >>>>> - Tushar.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message