apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <vro...@apache.org>
Subject Re: [Proposal] Simulate setting for application launch
Date Fri, 22 Dec 2017 16:19:44 GMT
I don't see more complexity in implementing a plugin compared to 
implementing an application. Additionally, for the use case you mention, 
plugin is a better option, as likely the behavior applies not to a 
single application, but all applications in that environment.

Websocket gateway address is a configuration parameter. A DAG may change 
depending on configuration parameters (presence of a gateway, hadoop 
version/vendor, security being enabled or disabled), but it should not 
change depending whether DAG is populated for a launch or to get an info.

Thank you,


On 12/21/17 10:05, Pramod Immaneni wrote:
> Asking users to create plugins for something they want to do in their
> application logic is to do things in an indirect and cumbersome way with an
> added level of complexity. I don't think users will elect to do that. There
> is a reason populateDAG and the operators give users the flexibility they
> do today to have any custom logic they want. populateDAG isn't only for
> returning a constant DAG for an application, the configuration that is
> passed today to populateDAG, apart from hadoop environmental properties
> that could be considered constant also includes a variable component, which
> is the user customizable configuration properties. There are already
> examples of applications that have used these properties to do something
> different. Apart from the properties, some attributes are also injected
> into the DAG in a deliberate fashion by the platform to provide user with
> these so they can create the dag accordingly. One example is a websocket
> gateway address. If this is present applications create a websocket output
> operator else they end up create a console or some other output operator.
> On Thu, Dec 21, 2017 at 8:27 AM, Vlad Rozov <vrozov@apache.org> wrote:
>> "Sometimes" is not a use case. Config is not a context.
>> Without concrete use cases the proposed change is not well justified.
>> populateDAG() is supposed to populate DAG, not to record anything in an
>> external system. It was a design goal for plugins.
>> Thank you,
>> Vlad
>> On 12/20/17 02:23, Priyanka Gugale wrote:
>>> +1
>>> Sometimes this context is required. We shouldn't change any default
>>> behaviour other than making this config available.
>>> -Priyanka
>>> On Wed, Dec 20, 2017 at 5:32 AM, Pramod Immaneni <pramod@datatorrent.com>
>>> wrote:
>>> The external system recording was just an example, not a specific use
>>>> case.
>>>> The idea is to provide comprehensive information to populateDAG as to the
>>>> context it is being called under. It is akin to the test mode or simulate
>>>> flag that you see with various utilities. The platform cannot control
>>>> what
>>>> populateDAG does, even without this information, in multiple calls that
>>>> you
>>>> mention the application can return different DAGs by depending on
>>>> any external factor such as time of day or some external variable. This
>>>> is
>>>> to merely provide more context information in the config. It is upto the
>>>> application to do what it wishes with it.
>>>> On Tue, Dec 19, 2017 at 2:28 PM, Vlad Rozov <vrozov@apache.org> wrote:
>>>> -0.5: populateDAG() may be called by the platform as many times as it
>>>>> needs (even in case it calls it only once now to launch an application).
>>>>> Passing different parameters to populateDAG() in simulate launch mode
>>>>> and
>>>>> actual launch may lead to different DAG being constructed for those two
>>>>> modes. Can't the use case you described be handled by a plugin?
>>>>> Thank you,
>>>>> Vlad
>>>>> On 12/19/17 10:06, Sanjay Pujare wrote:
>>>>> +1 although I prefer something that is more enforceable. So I like the
>>>>>> idea
>>>>>> of another method but that introduces incompatibility so may be in
>>>>>> On Tue, Dec 19, 2017 at 9:40 AM, Munagala Ramanath <
>>>>>> amberarrow@yahoo.com.invalid> wrote:
>>>>>>     +1
>>>>>>> Ram
>>>>>>>        On Tuesday, December 19, 2017, 8:33:21 AM PST, Pramod
Immaneni <
>>>>>>> pramod@datatorrent.com> wrote:
>>>>>>>     I have a mini proposal. The command get-app-package-info
runs the
>>>>>>> populateDAG method of an application to construct the DAG but
does not
>>>>>>> actually launch the DAG. An application developer does not know
>>>>>> which
>>>>> context the populateDAG is being called. For example, if they are
>>>>>>> recording
>>>>>>> application starts in an external system from populateDAG, they
>>>>>> have
>>>>> false entries there. This can be solved in different ways such as
>>>>>>> introducing another method in StreamingApplication or more parameters
>>>>>>> to populateDAG but a non disruptive option would be to add a
>>>>>> in
>>>>> the configuration object that is passed to populateDAG to indicate if
>>>>>> it
>>>>> is
>>>>>>> simulate/test mode or real launch. An application developer can
>>>>>> this
>>>>> property to take the appropriate actions.
>>>>>>> Thanks

View raw message