apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amol Kekre <a...@datatorrent.com>
Subject Re: Modules support in Apex
Date Thu, 03 Sep 2015 07:18:07 GMT
Atri,
For a lot of operations module should be treated as a black box. It is just
another reusable IP. The flattening should happen at launch time.

If we think of Apex as a compiler, then all the compile time checks (ports
connectivity, matching types/schema, properties, attributes, ...) are as
applicable to modules as to operators. At launch time (aka code generation
time) module gets flattened. Webservice should still enable access via
module scope on a running app.

Thks,
Amol


On Thu, Sep 3, 2015 at 12:10 AM, Atri Sharma <atri@apache.org> wrote:

> So the idea around our APEX-3 work will be that we will implement Module
> interface to build a class that adds operators at runtime? Sounds like a
> good idea, if Module is essentially a set of operators plugged in DAG.
>
> Is Module to be treated like a black box with input and output ports, and
> the internal subgraph either generated statically or dynamically?
>
> On Tue, Sep 1, 2015 at 2:47 AM, Vlad Rozov <v.rozov@datatorrent.com>
> wrote:
>
> > Atri,
> >
> > As a first cut module is a predefined subgraph that can be inserted into
> a
> > DAG. Generally speaking module is not required to expand to the same
> > subgraph. Depending on module properties or configuration, module may
> > expand to different subgraphs. At the same time, similar to an operator
> > module has predefined input/output ports and properties.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 8/31/15 11:46, Atri Sharma wrote:
> >
> >> No, I dont think APEX-3's functionality needs to be exceeded for this.
> >> What
> >> I am trying to understand here is the concept of Module. Is it a family
> of
> >> operators defined by same interface implementation *or* is it a defined
> >> subgraph that can be replaced and used as a part of a subgraph instead
> of
> >> building the whole connection again?
> >>
> >> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <tushar@datatorrent.com
> >
> >> wrote:
> >>
> >> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG, If
> >>> we
> >>> can
> >>> expand the DAG dynamically then we can use that functionality to expand
> >>> the
> >>> DAG
> >>> with know list of operator and connections between them. In a way
> APEX-3
> >>> provides
> >>> bigger functionality than APEX-55.
> >>>
> >>> Let me know if you think that module functionality requires more
> support
> >>> that what will be provided by APEX-3.
> >>>
> >>>
> >>>
> >>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <atri@apache.org> wrote:
> >>>
> >>> Looks interesting.
> >>>>
> >>>> Can you explain a bit on how it helps APEX-3? The objective of APEX-3
> is
> >>>>
> >>> to
> >>>
> >>>> have dynamic expansion of sub DAGs. If I understand correctly, the
> >>>> static
> >>>> list is to be known when declaring a module.
> >>>>
> >>>> Please correct me if I am wrong.
> >>>>
> >>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <
> tushar@datatorrent.com
> >>>> >
> >>>> wrote:
> >>>>
> >>>> Hi All,
> >>>>>
> >>>>> We are working on adding support for Modules in Apex. A module is
a
> >>>>>
> >>>> group
> >>>
> >>>> of operators that will have their own existence and will ease the way
> >>>>>
> >>>> we
> >>>
> >>>> are currently defining an application.
> >>>>>
> >>>>> A module is defined by:
> >>>>>
> >>>>>     - list of operators.
> >>>>>     - list of input and output ports.
> >>>>>     - set of properties for the module.
> >>>>>     - set of attributes for the module.
> >>>>>
> >>>>>
> >>>>> Details on the proposed work is given at:
> >>>>>
> >>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also
> related
> >>>>>
> >>>> to
> >>>>
> >>>>> https://malhar.atlassian.net/browse/APEX-3
> >>>>>
> >>>>> Regards,
> >>>>> -Tushar.
> >>>>>
> >>>>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message