apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Atri Sharma <a...@apache.org>
Subject Re: Modules support in Apex
Date Thu, 03 Sep 2015 08:12:26 GMT
Amol.

For my understanding, when you mention launch time/code generation time,
are you referring to generation of physical plan, please?

Regards,

Atri

On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre <amol@datatorrent.com> wrote:

> Atri,
> For a lot of operations module should be treated as a black box. It is just
> another reusable IP. The flattening should happen at launch time.
>
> If we think of Apex as a compiler, then all the compile time checks (ports
> connectivity, matching types/schema, properties, attributes, ...) are as
> applicable to modules as to operators. At launch time (aka code generation
> time) module gets flattened. Webservice should still enable access via
> module scope on a running app.
>
> Thks,
> Amol
>
>
> On Thu, Sep 3, 2015 at 12:10 AM, Atri Sharma <atri@apache.org> wrote:
>
> > So the idea around our APEX-3 work will be that we will implement Module
> > interface to build a class that adds operators at runtime? Sounds like a
> > good idea, if Module is essentially a set of operators plugged in DAG.
> >
> > Is Module to be treated like a black box with input and output ports, and
> > the internal subgraph either generated statically or dynamically?
> >
> > On Tue, Sep 1, 2015 at 2:47 AM, Vlad Rozov <v.rozov@datatorrent.com>
> > wrote:
> >
> > > Atri,
> > >
> > > As a first cut module is a predefined subgraph that can be inserted
> into
> > a
> > > DAG. Generally speaking module is not required to expand to the same
> > > subgraph. Depending on module properties or configuration, module may
> > > expand to different subgraphs. At the same time, similar to an operator
> > > module has predefined input/output ports and properties.
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > >
> > > On 8/31/15 11:46, Atri Sharma wrote:
> > >
> > >> No, I dont think APEX-3's functionality needs to be exceeded for this.
> > >> What
> > >> I am trying to understand here is the concept of Module. Is it a
> family
> > of
> > >> operators defined by same interface implementation *or* is it a
> defined
> > >> subgraph that can be replaced and used as a part of a subgraph instead
> > of
> > >> building the whole connection again?
> > >>
> > >> On Mon, Aug 31, 2015 at 11:27 PM, Tushar Gosavi <
> tushar@datatorrent.com
> > >
> > >> wrote:
> > >>
> > >> Yes, you are correct. The APEX-3 is for dynamic expansion of the DAG,
> If
> > >>> we
> > >>> can
> > >>> expand the DAG dynamically then we can use that functionality to
> expand
> > >>> the
> > >>> DAG
> > >>> with know list of operator and connections between them. In a way
> > APEX-3
> > >>> provides
> > >>> bigger functionality than APEX-55.
> > >>>
> > >>> Let me know if you think that module functionality requires more
> > support
> > >>> that what will be provided by APEX-3.
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Aug 31, 2015 at 10:39 AM, Atri Sharma <atri@apache.org>
> wrote:
> > >>>
> > >>> Looks interesting.
> > >>>>
> > >>>> Can you explain a bit on how it helps APEX-3? The objective of
> APEX-3
> > is
> > >>>>
> > >>> to
> > >>>
> > >>>> have dynamic expansion of sub DAGs. If I understand correctly,
the
> > >>>> static
> > >>>> list is to be known when declaring a module.
> > >>>>
> > >>>> Please correct me if I am wrong.
> > >>>>
> > >>>> On Mon, Aug 31, 2015 at 11:05 PM, Tushar Gosavi <
> > tushar@datatorrent.com
> > >>>> >
> > >>>> wrote:
> > >>>>
> > >>>> Hi All,
> > >>>>>
> > >>>>> We are working on adding support for Modules in Apex. A module
is a
> > >>>>>
> > >>>> group
> > >>>
> > >>>> of operators that will have their own existence and will ease the
> way
> > >>>>>
> > >>>> we
> > >>>
> > >>>> are currently defining an application.
> > >>>>>
> > >>>>> A module is defined by:
> > >>>>>
> > >>>>>     - list of operators.
> > >>>>>     - list of input and output ports.
> > >>>>>     - set of properties for the module.
> > >>>>>     - set of attributes for the module.
> > >>>>>
> > >>>>>
> > >>>>> Details on the proposed work is given at:
> > >>>>>
> > >>>>> https://malhar.atlassian.net/browse/APEX-55 . The work is also
> > related
> > >>>>>
> > >>>> to
> > >>>>
> > >>>>> https://malhar.atlassian.net/browse/APEX-3
> > >>>>>
> > >>>>> Regards,
> > >>>>> -Tushar.
> > >>>>>
> > >>>>>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message