apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: AppFactory in Apex
Date Wed, 20 Apr 2016 05:57:21 GMT
#2 JSON format is already supported.

#4 is an example how you can translate a specification into a logical
plan without any change in core.

I think it is important to discuss this in the context of a use case.

Thomas

On Mon, Apr 18, 2016 at 10:42 PM, Chinmay Kolhatkar <chinmay@apache.org> wrote:
> Hi Thomas,
>
> Can you please elaborate on "different DAG specification can be translated
> into a Logical Plan using existing API"?
>
> I don't have specific usecase in mind. This is just an idea that I wanted
> to share with the community on following thought process:
>
> 1. With current AppFactory impl (for JSON, properties file and
> populateDAG), gives us those 3 ways to construct and configure DAG. Lets
> say one wants to have own DAG specification format, then that's not
> possible as of now.
>
> 2. JSON format opens a new way of creating DAG. To me, this should a great
> power of apex engine that DAG specification and configuration is not
> limited to Java code, but even more. Making a configurable AppFactory can
> open discussion to even better formats/standards for specifications of DAG.
>
> 3. If JSON/Properties file format needs a change either in DAG
> specification/configuration, this means, the changes has to go in Apex
> core. Instead, if we allow overriding AppFactory, one could have more DAG
> specifications/conf added to Malhar making lesser changes in apex
> core/engine. I'm thinking more in perspective of stability of apex engine
> release build.
>
> 4. Recently Siyuan shared high level APIs for apex application. Having
> a separate AppFactory for such high level APIs can make this more modular.
>
> 5. Another example where this could be useful is Apex DataFrames shared by
> Capital One folks.
>
> Overall, this is just an idea and is not bound by any specific usecase.
> The basic idea is
> - Plug-ability to apex engine
> - Make apex platform more extensible
> - Reducing configuration/DAG specification related changes making
> apex engine more sturdy.
>
>
> Thanks,
> Chinmay.
>
>
>
> On Tue, Apr 19, 2016 at 7:56 AM, Thomas Weise <thomas@datatorrent.com>
> wrote:
>
>> Chinmay,
>>
>> Different DAG specifications can be translated into a logical plan
>> using the existing API.
>>
>> AppFactory is an implementation detail. Can you describe the problem
>> you are looking to solve?
>>
>> Thanks
>>
>>
>> On Mon, Apr 18, 2016 at 6:21 PM, Chinmay Kolhatkar <chinmay@apache.org>
>> wrote:
>> > Dear Community,
>> >
>> > Does apex engine allows user to override or extend existing
>> implementation
>> > of AppFactory from outside of engine?
>> >
>> > If it not, would allowing it make any required launch time enhancements
>> > easier and safer (for stability) rather than changing apex engine code?
>> >
>> > Another advantage I see by doing this is, user can have their own DAG
>> > specification format if required.
>> >
>> > Thanks,
>> > Chinmay.
>>

Mime
View raw message