apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tushar Gosavi <tus...@datatorrent.com>
Subject Re: [APEX-107] Support for specifying module properties on modules.
Date Wed, 02 Dec 2015 18:16:34 GMT
Thanks Vlad, I will check if we can override addOperator in DAG to accept
Module too. This way we can have similar API.

- Tushar.


On Tue, Dec 1, 2015 at 11:54 PM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> We should either provide Java API that supports interchangeability between
> operators and modules or ignore how an operator/module was added to a DAG.
>
> Thank you,
>
> Vlad
>
>
> On 11/30/15 20:21, Tushar Gosavi wrote:
>
>> Good point Vlad, if the class implements both Operator and Module
>> interface. We will consider it as a module by default in json API, But
>> also
>> provide a way to specify his intention whether it should be considered as
>> Module or Operator.
>>
>> If user use a Java API then intention is clear whether the object should
>> be
>> considered as Module or Operator (addOperator v/s addModule), will it be
>> correct to change the intention of the user internally based on the type
>> of
>> component?
>>
>> - Tushar.
>>
>>
>> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <v.rozov@datatorrent.com>
>> wrote:
>>
>> +1. Modules and operators should be interchangeable. It should be possible
>>> for an operator designer to replace it with a module without requirement
>>> to
>>> recompile all applications that use the existing operator. From
>>> application
>>> designer view both operators and modules may be considered as a black box
>>> with a private implementation that may be monolithic (operator) or
>>> distributed (module).
>>>
>>> Note that a Java class may implement both Operator and Module interface
>>> and be added to a DAG using addOperator() method. At run-time (Logical
>>> Plan->Physical Plan) Apex platform should handle such classes as modules
>>> and may not depend on how a module was added to a DAG, IMO.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 11/30/15 08:41, Tushar Gosavi wrote:
>>>
>>> addOperator method in DAG interface takes object of type Operator, unless
>>>> we make Module as subclass of Operator we can not use the same API.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <thomas@datatorrent.com>
>>>> wrote:
>>>>
>>>> +1
>>>>
>>>>> If there are really module specific constructs, they can be added
>>>>> later.
>>>>>
>>>>> Why is the same not supported in the Java API, i.e. why not allow a
>>>>> module
>>>>> to be added like an operator and handle the distinction in the
>>>>> implementation?
>>>>>
>>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <tushar@datatorrent.com
>>>>> >
>>>>> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>>> I am going to add a feature to allow use of modules in property file
>>>>>> and
>>>>>> json api.
>>>>>> I am planing to use same "operators" array for adding modules as
well
>>>>>> as
>>>>>> operators. The reasoning behind is that user don't have to worry
about
>>>>>>
>>>>>> the
>>>>>
>>>>> component being added to the DAG is actually an operator or a module,
>>>>>> as
>>>>>> the method to specify their properties and to create instance of
them
>>>>>> is
>>>>>> same. And it results in less complex syntax for writing application
>>>>>> using
>>>>>> json/property file.
>>>>>>
>>>>>> For example the sample Application specified using Json will look
like
>>>>>> below.
>>>>>> ```json
>>>>>> {
>>>>>>     "operators": [
>>>>>>       {
>>>>>>         "name": "operator1",
>>>>>>         "class": "com.datatorrent.lib.operator.Input",
>>>>>>         "properties": {
>>>>>>           "property1": "value1"
>>>>>>         }
>>>>>>       },
>>>>>>       {
>>>>>>         "name": "module1",
>>>>>>         "class": "com.datatorrent.module.Module1",
>>>>>>         "properties": {
>>>>>>           "property1": "value1"
>>>>>>         }
>>>>>>       },
>>>>>>     ],
>>>>>>     "streams": [
>>>>>>       {
>>>>>>         "name": "s1",
>>>>>>         "source": {
>>>>>>           "operatorName": "operator1",
>>>>>>           "portName": "output"
>>>>>>         },
>>>>>>         "sinks": [
>>>>>>           {
>>>>>>             "operatorName": "module",
>>>>>>             "portName": "input"
>>>>>>           }
>>>>>>         ]
>>>>>>       }
>>>>>>     ]
>>>>>> }
>>>>>> ```
>>>>>>
>>>>>> In above example "module1" is of type Module. While processing the
>>>>>> Json
>>>>>>
>>>>>> if
>>>>>
>>>>> the
>>>>>> class of component is of type Module then we will use addModule api
to
>>>>>>
>>>>>> the
>>>>>
>>>>> DAG.
>>>>>>
>>>>>> Let me know, what do you think about this.
>>>>>>
>>>>>> Regards,
>>>>>> - Tushar.
>>>>>>
>>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <
>>>>>> tushar@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>> A module is superset of operator before expansion in logical plan,
>>>>>> except
>>>>>> operator
>>>>>>
>>>>>>> attribute are not applicable to modules. As Tim suggested we
will
>>>>>>> have
>>>>>>>
>>>>>>> to
>>>>>> add checks
>>>>>>
>>>>>>> later to validate these attributes settings, and not allow top
level
>>>>>>> operator and
>>>>>>> module to share same name.
>>>>>>>
>>>>>>> The same question comes when we want to add module to application
>>>>>>> using
>>>>>>> property
>>>>>>> file API and JSON file API.
>>>>>>>
>>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator>
>>>>>>>
>>>>>>> One approach we can take is, if class is of type module, then
use
>>>>>>> addModule api
>>>>>>> to add module to the dag, else use addOperator api. The
>>>>>>> initialization
>>>>>>>
>>>>>>> of
>>>>>> object and applying property is generic can be reused by module.
>>>>>>
>>>>>>> Similarly we have Json API where all operators are specified
under
>>>>>>> operators array
>>>>>>>
>>>>>>> {
>>>>>>>     "operators": [
>>>>>>>       {.. operators .. }
>>>>>>>     ],
>>>>>>>     "streams": [
>>>>>>>       {.. streams .. }
>>>>>>>     ]
>>>>>>> }
>>>>>>>
>>>>>>> We could add module in the operators array, and based on the
object
>>>>>>>
>>>>>>> type
>>>>>> we can
>>>>>>
>>>>>>> consider it as module or operator. The mechanism of initializing
>>>>>>> object
>>>>>>> and applying
>>>>>>> properties remains same.
>>>>>>>
>>>>>>> - Tushar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message