apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Modules
Date Tue, 03 Nov 2015 17:45:31 GMT
For 7 does the current implementation allow module properties to be set
from a property file.

Thanks

On Tue, Nov 3, 2015 at 4:14 AM, Sandeep Deshmukh <sandeep@datatorrent.com>
wrote:

> 1. Agree. We will address this in the current pull request.
>
> 2. The idea behind keeping it that way was to keep it simple and minimum
> learning curve for Module users. Module is focused on a functionality and
> an application would be typically consisting of few modules, extreme case,
> just one. Modules is in between an operator and an application.
>
> 3. We do not want to expose internals of the module to the module user.
> Given this, we will refrain from exposing any operator or its properties to
> the module user.
>
> 4. Same as 3 above. No direct access to operators as well as attributes.
> The operator attributes that needs to be exposed, are accessible module
> properties. Module designer has to explicitly provide properties for this.
>
> 5. Module specific properties are already taken care of. Module attributes
> work will be taken up later.
>
> 6. Agreed. As we only have module level properties and they are propagated
> by the module internally, I think we have already achieved this in Modules.
> For application level properties, we would also need to take similar
> approach.
>
> 7. For module properties, we have it in place. For module attributes, the
> work will be taken up later.
>
> 8. To be taken up later.
>
> In general, the modules work is split into stages where we would like
> deliver core part which can unblock module designers and then take next
> stage of work. Currently we are at first stage. The next stage will get
> into advanced topics like module attributes and partitioning.
>
> We attempt to inherit ease of use and simplicity of Apex into Modules. This
> has lead to reusing function names as populateDAG. After all,  Shakespeare
> said " What's in a name?"  :)
>
> Regards,
> Sandeep
>
> On Tue, Nov 3, 2015 at 6:30 AM, Amol Kekre <amol@datatorrent.com> wrote:
>
> > 3,4 should follow conventions where the creator decides the api
> (including
> > accessibility). In general only those properties exposed by module
> creator
> > should be settable. What the module internally does with them is module
> > designer's call. Accessing internals of module from outside is uncommon.
> > For exampe in Java (or C++) private fields/members are not to be
> accessed.
> > Properties (setter and getter) are the api that module designer gives to
> > the module user. It is dangerous and has unintended consequences if
> module
> > user starts to access internals outside the api.
> >
> > Partitioning should be next phase. As long as current design does not
> halt
> > partitioning it should be ok (which I believe is true).
> >
> > Thks,
> > Amol
> >
> >
> > On Mon, Nov 2, 2015 at 3:44 PM, Vlad Rozov <v.rozov@datatorrent.com>
> > wrote:
> >
> > > 1. +1, though passing original DAG to module's populateDAG() it is not
> by
> > > design and is the current pull request implementation details.
> > >
> > > 2. While I agree that both Module and StreamingApplication let's
> > > module/application designer to expose DAG design reuse pattern and
> > > StreamingApplication interface may be extending Module, it does not
> seem
> > to
> > > buy us much. Do we want to allow certain applications to be reused as
> > > Modules in other applications or should application package be
> different
> > > from Module package? The current approach is to distribute Modules as
> > part
> > > of .jar for example as part of Malhar library without necessarily
> > providing
> > > all necessary dependencies. Application package on other side must
> > include
> > > all dependencies not provided by the platform.
> > >
> > > 3, 4. While this will help Module designer, it may complicate Module
> > > maintenance and how Modules are used. What if Module designer wants to
> > > change it's implementation and replace one operator implementation with
> > > another operator? Does StreamingApplication designer need to know
> > internal
> > > structure of Modules? Should Module be considered as a black box during
> > > Application design time as it was initially planned?
> > >
> > > 5, 6, 7 +1. This is currently proposed behavior of Module functionality
> > > the way I understand it.
> > >
> > > 8. We need to see what Module designer can specify for partitioning.
> One
> > > of supported cases should be ability to specify cascading partitioning
> > > scheme.
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > >
> > > On 11/2/15 10:30, Pramod Immaneni wrote:
> > >
> > >> I have some comments and suggestions on the module design. I think
> these
> > >> need to be taken into account before we can merge the implementation
> > >> provided below into the mainline code. I apologize if these should
> have
> > >> been brought up earlier as for some reason or the other I was out of
> > loop
> > >> on this one
> > >>
> > >>      https://github.com/apache/incubator-apex-core/pull/148
> > >> <
> > >>
> >
> https://github.com/apache/incubator-apex-core/pull/148#issuecomment-153104963
> > >> >
> > >>
> > >>      1. DAG scoping currently in the implementation is global for
> > modules,
> > >> each module's populateDAG sees the entire DAG. It should be locally
> > scoped
> > >> as one module does not and should not know about another.
> > >>      2. The module has a populateDAG method with exact same syntax as
> in
> > >> StreamingApplication. Is StreamingApplication also a module, should it
> > >> extend that interface.
> > >>      3. Setting properties for modules is too verbose. Module
> developer
> > >> needs to repeat every property they want exposed with a setter and
> > getter
> > >> in JAVA. I don't disagree that module developer should be able to
> choose
> > >> which properties from which operators need to be exposed but the
> current
> > >> way seems to duplicate code. Here is a suggestion.
> > >>           a. Allow modules to specify which operators and properties
> can
> > >> be
> > >> accessible from outside. One way is in the "populateDAG" method of the
> > >> module when adding the operator have the ability to specify if this
> > >> operator can be accessible from outside and which or all properties
> can
> > be
> > >> accessible.
> > >>           b. Provide methods in ModuleMeta or elsewhere to set
> property
> > >> values by specifying the operator name (friendly name) inside the
> module
> > >> and property name. If this is allowed by a. above it is successful
> else
> > it
> > >> should fail.
> > >>           c. Allow a syntax in property files to specify the property
> in
> > >> b.
> > >> Example syntax dt.module.<modulename>.operator.<operatorname>.prop.<
> > >> propname>
> > >>      4. For attributes same mechanism as in 3 should apply for the
> > >> operators
> > >> that are exposed by the module.  For property file, example syntax
> > >> dt.module.<modulename>.operator.<operatorname>.attr.<attrname>
> > >>      5. Module developers in addition to 3. and 4. above may choose to
> > >> support module level properties and attributes. These should not be
> the
> > >> default when 3. and 4. are possible but complementary, in addition to
> > >> them.
> > >> In this case for properties they can implement setters and getters in
> > the
> > >> module. For attributes the user should still be able to set the
> > attributes
> > >> using the dag setAttribute method. You could introduce a method in the
> > >> module to process attributes that can get called by the engine once
> > >> everything is set.
> > >>      6. For 5. above setting global properties and attributes for
> module
> > >> is
> > >> akin to ideas that have been proposed for the application as well. A
> > >> consistent way must be possible for applications as well even if it is
> > not
> > >> implemented now.
> > >>      7. For 5. or 6. above there should be a property file way of
> > >> specifying
> > >> the global module properties and attributes. Example syntax
> > >> dt.module.<modulename>.prop.<propname>,
> > >> dt.module.<modulename>.attr.<attrname>.
> > >> Notice the difference with 3. c. and 4 above that there is no operator
> > >> keyword here.
> > >>      8. Partitioning needs to be consistent with what the user will
> > expect
> > >> when they see module as an entity. I will send an image of possible
> > >> examples of how the user will expect the physical plan to look in
> > certain
> > >> cases.
> > >>
> > >> Thanks
> > >>
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message