apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@apache.org>
Subject Re: AppFactory in Apex
Date Tue, 19 Apr 2016 05:42:53 GMT
Hi Thomas,

Can you please elaborate on "different DAG specification can be translated
into a Logical Plan using existing API"?

I don't have specific usecase in mind. This is just an idea that I wanted
to share with the community on following thought process:

1. With current AppFactory impl (for JSON, properties file and
populateDAG), gives us those 3 ways to construct and configure DAG. Lets
say one wants to have own DAG specification format, then that's not
possible as of now.

2. JSON format opens a new way of creating DAG. To me, this should a great
power of apex engine that DAG specification and configuration is not
limited to Java code, but even more. Making a configurable AppFactory can
open discussion to even better formats/standards for specifications of DAG.

3. If JSON/Properties file format needs a change either in DAG
specification/configuration, this means, the changes has to go in Apex
core. Instead, if we allow overriding AppFactory, one could have more DAG
specifications/conf added to Malhar making lesser changes in apex
core/engine. I'm thinking more in perspective of stability of apex engine
release build.

4. Recently Siyuan shared high level APIs for apex application. Having
a separate AppFactory for such high level APIs can make this more modular.

5. Another example where this could be useful is Apex DataFrames shared by
Capital One folks.

Overall, this is just an idea and is not bound by any specific usecase.
The basic idea is
- Plug-ability to apex engine
- Make apex platform more extensible
- Reducing configuration/DAG specification related changes making
apex engine more sturdy.


On Tue, Apr 19, 2016 at 7:56 AM, Thomas Weise <thomas@datatorrent.com>

> Chinmay,
> Different DAG specifications can be translated into a logical plan
> using the existing API.
> AppFactory is an implementation detail. Can you describe the problem
> you are looking to solve?
> Thanks
> On Mon, Apr 18, 2016 at 6:21 PM, Chinmay Kolhatkar <chinmay@apache.org>
> wrote:
> > Dear Community,
> >
> > Does apex engine allows user to override or extend existing
> implementation
> > of AppFactory from outside of engine?
> >
> > If it not, would allowing it make any required launch time enhancements
> > easier and safer (for stability) rather than changing apex engine code?
> >
> > Another advantage I see by doing this is, user can have their own DAG
> > specification format if required.
> >
> > Thanks,
> > Chinmay.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message