Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAD0KA=Pg0AT1qsvh_R7aVN6iVWv-88JO0ncq64qgVupa7sJ4pw@mail.gmail.com>
References: 
 <CAD0KA=Pg0AT1qsvh_R7aVN6iVWv-88JO0ncq64qgVupa7sJ4pw@mail.gmail.com>
Date: Mon, 28 Mar 2016 22:31:29 +0530
Message-ID: 
 <CABAipVbE6hQpn6Epi4U_FC-wXpuqW7-BEap5NTrb1wNp9qerVg@mail.gmail.com>
Subject: Re: [APEXCORE-107] Adding modules in DAG through property and json
 file.
From: Chinmay Kolhatkar <chinmay@apache.org>
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c130ba278d7e052f1edd04

--001a11c130ba278d7e052f1edd04
Content-Type: text/plain; charset=UTF-8

+1 for approach 2.

---
Sent from mobile.
On 23 Mar 2016 4:25 p.m., "Tushar Gosavi" <tushar@datatorrent.com> wrote:

> Hi All,
>
> I am planning provide support for adding modules into the json and property
> file specification of DAG. We will go with the same syntax as discussed in
> following mail thread.
>
>
> https://mail-archives.apache.org/mod_mbox/incubator-apex-dev/201512.mbox/%3C565D0E2C.2000805%40datatorrent.com%3E
>
>
> There are multiple choices for the design and I want some suggestion from
> the community aboutg how to go about adding this support. To support
> json/property format, Module should support following functionality
> - Property format uses setSource, addSink methods of StreamMeta, which were
> not properly supported by Module.
> - Module need capability of extracting port object given names. Operator
> provides this functionality through Operators.describe.
>
> Approach 1)
> As module meta and operator meta shares common fields such as name, port
> information. We can separate this out in a common class NodeMeta and derive
> OperatorMeta and ModuleMeta from it. This class will handle extracting port
> information form the object. The changes involve are
>     - Split OperatorMeta object
>     - PortMeta objects will contain NodeMeta references than OperatorMeta,
> in some places we will have to perform unchecked cast from NodeMeta to
> OperatorMeta. where code was expecting OperatorMeta from the
> PortMeta.
>     - Support setSource and addSink methods.
>     - Change Operators.describe to accept Module or Operator and return
> port mapping.
>     - Need instanceof call at few places to check if object is Module or
> Operator, before calling specific API while working with property and json
> file.
>     - change signature of few method which accepts Operator to Object. as
> Module and Operator do not share a common parent but requires
> common processing
>     - Replace OperatorMeta with NodeMeta in most of the classes.
>
>
> Approach 2)
> Make Module extends Operator and make ModuleMeta extends OperatorMeta. The
> changes involved will be
>  - Change Module interface to extend Operator
>  - add support for setSource and addSink for Modules.
>  - addOperator will inspect type of object and call addModule if operator
> is a module.
>
> Disadvantage
> - This will break compatibility but this should not be a problem as Module
> is still an evolving API.
> - Operator lifecyle methods will not get executed for module and we can
> document this.
>
> Advantages
> - This will avoid code duplication at multiple places where there is
> similarity between Module and Operator, and there is lot of similarity
> while defining logical DAG.
> - This will also automatically add support for Module in api being
> developed for operator. For example high level api.
> - This will make module and operator interchangeable in application.
>
> I will prefer Approach 2.
>
> Regards,
> -Tushar.
>

--001a11c130ba278d7e052f1edd04--