falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth Sundarrajan <srik...@hotmail.com>
Subject RE: [DISCUSS] Namespace-ing properties and FALCON-1573
Date Fri, 20 Nov 2015 09:30:15 GMT
Thanks Pallavi for bring this up. I have been wanting namespace isolation for properties for
a fair while and this allows to build consensus and move forward.

>From what I can gather from the discussions on 1573, multiple issues are hinted at and
proposed to be solved.

1. Allowing for properties being overidden at schedule time - directly being passed to underlying
scheduler, essentially a pass through for falcon
2. Allowing for default properties in falcon configured in entity definition to be overriden
at schedule time
3. Namespacing of properties such that there is no accidental clash in property name and there
is general improvement in maintainability.

To pass properties that override falcon entity “properties” and have them passed through
as is to the underlying scheduler would be a good start. 

My personal preference would be for the following overlay scheme, where user overrides gets
preference and the resolution order is as follows.

——> FALCON SYSTEM <— — 
(USER PROPERTIES)
  > (PROCESS PROPERTIES)
    > (FEED PROPERTIES)
      > (CLUSTER PROPERTIES)

——> SCHEDULER SYSTEM (OOZIE) <— — 
@Schedule / Instance Management 
(RESOLVED PROPERTIES FROM ABOVE)
  > (SCHEDULER DEFAULT PROPERTIES)

The properties in the entity definition today aren’t totally opaque (for ex. queue-name,
priority etc). We ought to segregate properties into various classes such as System level
Internal properties, Entity level internal properties, Entity level properties, Entity level
user properties. 

System level internal properties are used by falcon and is application to the entire instance
and as such isn’t available for override. However these properties can be re-read by the
server if it is deemed to be a runtime property. Tucking them under a namespace is desirable
(for. ex falcon.system). Properties with this namespace if sent for any entity or instance
operation should be ignored and appropriately warned.

Entity level internal property is what is used by falcon internally to exchange information
between different systems and these shouldn’t overridable either. Namespace for these could
be say falcon.internal. Properties with this namespace if sent for any entity or instance
operation should be ignored and appropriately warned.

Entity level properties can be overriden by users. Seems very important to put this in an
exclusive namespace (for ex. falcon.entity.override). This is well defined list of properties
and by definition is not a first class property element in the property bag of entity. For
ex. priority, process-parallelism, scheduler etc

Entity level user properties are the ones typically enlisted in the property bag of the entity
definition. Anything supplied by the user overrides or is additionally added to the entity
or instance operation. This I would prefer to keep it without any namespace, as these property
names make it ways into workflow definition and possibly into user code and it would be preferable
not to carry the prefix all the way and muddy user code.

Optionally if we particularly feel the need to pass something to the scheduler and scheduler
only, we can set aside a namespace for it.

Regards
Srikanth Sundarrajan

> Date: Mon, 16 Nov 2015 10:02:57 +0530
> Subject: [DISCUSS] Namespace-ing properties and FALCON-1573
> From: pallavi.rao@inmobi.com
> To: dev@falcon.apache.org
> CC: dadelcas@hotmail.com
> 
> Folks,
> With FALCON-1573, Daniel came up with a very good suggestion of allowing
> user-supplied properties at schedule time. He proposed 2 approaches
> <https://issues.apache.org/jira/browse/FALCON-1573?focusedCommentId=14982386&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14982386>
> -
> 
>    1. Allow user to specify free form name-value pairs. Since user-supplied
>    property names can clash with Falcon property names, give Falcon properties
>    (generated when the bundle is built) priority over user-supplied ones, in
>    case of clash.
>    2. Namespace user-supplied properties, so we can distinguish user and
>    system properties. This will allow user to supply properties without any
>    clash with system properties.
> 
> Daniel has already uploaded patch for approach (1). Regarding this, wish to
> solicit your input on 2 questions:
> 
>    1. Should we pursue approach (2) in FALCON-1573 itself or take it up
>    separately?
>    2. When we eventually namespace the properties, what should the
>    namespace format be?
> 
> My 2 cents:
> 
> Question (1) : I think we should pursue namespace approach in FALCON-1573
> itself. Two reasons for the same:
> 
>    - When user supplies a name-value pair and we silently ignore (we only
>    log) it when it clashes with system property, it is not very intuitive to
>    the user.
>    - If we don't introduce it now, but, bring it in later, we'll have to
>    deal with 2 kinds of properties, one that is namespace-d and one that isn't.
> 
> Question (2) : I suggest we use falcon.system.* (or just falcon.*) for
> system properties and falcon.user.* for user-supplied properties.
> 
> Apologies for the tad lengthy mail. Request you all to provide your inputs.
> 
> Thanks and Regards,
> Pallavi
> 
> -- 
> _____________________________________________________________
> The information contained in this communication is intended solely for the 
> use of the individual or entity to whom it is addressed and others 
> authorized to receive it. It may contain confidential or legally privileged 
> information. If you are not the intended recipient you are hereby notified 
> that any disclosure, copying, distribution or taking any action in reliance 
> on the contents of this information is strictly prohibited and may be 
> unlawful. If you have received this communication in error, please notify 
> us immediately by responding to this email and then delete it from your 
> system. The firm is neither liable for the proper and complete transmission 
> of the information contained in this communication nor for any delay in its 
> receipt.
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message