apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: [APEX-107] Support for specifying module properties on modules.
Date Mon, 30 Nov 2015 16:08:54 GMT
Hi All,

I am going to add a feature to allow use of modules in property file and
json api.
I am planing to use same "operators" array for adding modules as well as
operators. The reasoning behind is that user don't have to worry about the
component being added to the DAG is actually an operator or a module, as
the method to specify their properties and to create instance of them is
same. And it results in less complex syntax for writing application using
json/property file.

For example the sample Application specified using Json will look like
below.
```json
{
  "operators": [
    {
      "name": "operator1",
      "class": "com.datatorrent.lib.operator.Input",
      "properties": {
        "property1": "value1"
      }
    },
    {
      "name": "module1",
      "class": "com.datatorrent.module.Module1",
      "properties": {
        "property1": "value1"
      }
    },
  ],
  "streams": [
    {
      "name": "s1",
      "source": {
        "operatorName": "operator1",
        "portName": "output"
      },
      "sinks": [
        {
          "operatorName": "module",
          "portName": "input"
        }
      ]
    }
  ]
}
```

In above example "module1" is of type Module. While processing the Json if
the
class of component is of type Module then we will use addModule api to the
DAG.

Let me know, what do you think about this.

Regards,
- Tushar.

On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <tushar@datatorrent.com>
wrote:

>
> A module is superset of operator before expansion in logical plan, except
> operator
> attribute are not applicable to modules. As Tim suggested we will have to
> add checks
> later to validate these attributes settings, and not allow top level
> operator and
> module to share same name.
>
> The same question comes when we want to add module to application using
> property
> file API and JSON file API.
>
> dt.operator.<operatorname>.classname=<fqcn of operator>
>
> One approach we can take is, if class is of type module, then use
> addModule api
> to add module to the dag, else use addOperator api. The initialization of
> object and applying property is generic can be reused by module.
>
> Similarly we have Json API where all operators are specified under
> operators array
>
> {
>   "operators": [
>     {.. operators .. }
>   ],
>   "streams": [
>     {.. streams .. }
>   ]
> }
>
> We could add module in the operators array, and based on the object type
> we can
> consider it as module or operator. The mechanism of initializing object
> and applying
> properties remains same.
>
> - Tushar.
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message