apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: AppFactory in Apex
Date Wed, 20 Apr 2016 09:07:08 GMT
Chinmay,

I think what you are saying is why only specifically support json in the
engine (other than StreamingApplication and property builder of course and
I am including apex cli as part of the engine here). If json has been given
first class support as opposed to doing it outside the engine why not make
it more general to include others by making the translator (app factory) a
plugin into the engine and json support could be re-casted into a plugin. I
don't see why that is something we cannot do given that we already opened
the pandora's box with json.

Thanks

On Tue, Apr 19, 2016 at 10:57 PM, Thomas Weise <thomas@datatorrent.com>
wrote:

> #2 JSON format is already supported.
>
> #4 is an example how you can translate a specification into a logical
> plan without any change in core.
>
> I think it is important to discuss this in the context of a use case.
>
> Thomas
>
> On Mon, Apr 18, 2016 at 10:42 PM, Chinmay Kolhatkar <chinmay@apache.org>
> wrote:
> > Hi Thomas,
> >
> > Can you please elaborate on "different DAG specification can be
> translated
> > into a Logical Plan using existing API"?
> >
> > I don't have specific usecase in mind. This is just an idea that I wanted
> > to share with the community on following thought process:
> >
> > 1. With current AppFactory impl (for JSON, properties file and
> > populateDAG), gives us those 3 ways to construct and configure DAG. Lets
> > say one wants to have own DAG specification format, then that's not
> > possible as of now.
> >
> > 2. JSON format opens a new way of creating DAG. To me, this should a
> great
> > power of apex engine that DAG specification and configuration is not
> > limited to Java code, but even more. Making a configurable AppFactory can
> > open discussion to even better formats/standards for specifications of
> DAG.
> >
> > 3. If JSON/Properties file format needs a change either in DAG
> > specification/configuration, this means, the changes has to go in Apex
> > core. Instead, if we allow overriding AppFactory, one could have more DAG
> > specifications/conf added to Malhar making lesser changes in apex
> > core/engine. I'm thinking more in perspective of stability of apex engine
> > release build.
> >
> > 4. Recently Siyuan shared high level APIs for apex application. Having
> > a separate AppFactory for such high level APIs can make this more
> modular.
> >
> > 5. Another example where this could be useful is Apex DataFrames shared
> by
> > Capital One folks.
> >
> > Overall, this is just an idea and is not bound by any specific usecase.
> > The basic idea is
> > - Plug-ability to apex engine
> > - Make apex platform more extensible
> > - Reducing configuration/DAG specification related changes making
> > apex engine more sturdy.
> >
> >
> > Thanks,
> > Chinmay.
> >
> >
> >
> > On Tue, Apr 19, 2016 at 7:56 AM, Thomas Weise <thomas@datatorrent.com>
> > wrote:
> >
> >> Chinmay,
> >>
> >> Different DAG specifications can be translated into a logical plan
> >> using the existing API.
> >>
> >> AppFactory is an implementation detail. Can you describe the problem
> >> you are looking to solve?
> >>
> >> Thanks
> >>
> >>
> >> On Mon, Apr 18, 2016 at 6:21 PM, Chinmay Kolhatkar <chinmay@apache.org>
> >> wrote:
> >> > Dear Community,
> >> >
> >> > Does apex engine allows user to override or extend existing
> >> implementation
> >> > of AppFactory from outside of engine?
> >> >
> >> > If it not, would allowing it make any required launch time
> enhancements
> >> > easier and safer (for stability) rather than changing apex engine
> code?
> >> >
> >> > Another advantage I see by doing this is, user can have their own DAG
> >> > specification format if required.
> >> >
> >> > Thanks,
> >> > Chinmay.
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message