stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lakmal Warusawithana <lak...@wso2.com>
Subject Re: [Discuss] Grouping of services (cartridges)
Date Thu, 01 May 2014 08:17:56 GMT
On Thu, May 1, 2014 at 1:43 PM, Shaheed Haque <shahhaqu@cisco.com> wrote:

>  Hi,
>
>
>
> Sorry for top-posting, but one additional thing that I feel should be
> considered is a minimal set of lifecycle events for the group; for example:
>
>
>
> - "all members of a group are now active", aka "group active".
>
>
>
> - "at least one member of an existing group has failed", aka "group down".
>
>
>
> This is clearly an add-on to the core functionality but is needed to round
> the feature out.
>
>
>

+1


> Thanks, Shaheed
>
>
>
>
>
> On Thursday 01 May 2014 07:46:46 Lakmal Warusawithana wrote:
>
> Hi Martin,
>
>
> With the current implementation I think and agree its better to go with
> json as defining configuration data. Summarizing proposal and the
> discussion I can think following data model. Here I am thinking
> implementing single configuration data (application deployment definition)
> model, which can define full configuration of an composite application. we
> should take it down to several milestones. we can start on enhancing
> backend services in initial milestone and later will come with aggregating
> them. All of your thoughts welcome.
>
>
> Can you come up with some definition format. (may be json format). Also
> see my inline comment.
>
>
> On Wed, Apr 30, 2014 at 11:03 PM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi Lakmal,
>
>
>
> Below are ideas on how to break up the feature into a high level ToDo list
> (with some suggestions and questions) :
>
>
>
> + Defining configuration data format – for example json or chef / recipe
>    ++ would using recipes affect all configuration data or is it
> restricted to the grouping / dependency configuration only ?
>
>
> IMHO, its may easy with json, considering current existing services.
>
>
>
>
> + Adding the new model to stratos manager and integrate with existing
> model:
>
>    ++ Group model (nested groups, group properties)
>     ++ Dependency model
>     ++ in the current model we have a Subscriptions and Deployment
> manager, I think we need something similar to handle the groups which in
> turn invokes deployment / subscriptions manager
>
>
>
> +1
>
>
>
> + Defining runtime dependency conditions:
>
>       ++ Calculating the dependency for a cartridge (service) dynamically
> by walking the dependency tree and checking the state of dependencies (e.g.
> member state)
>
>
>
>
> +1
>
>
>
> + Integrate the dynamic aspect of the dependency model (checking the
> conditions of dependencies and act accordingly):
>       ++ As of now I think it has to be integrated with the autoscaler as
> it the entity which controls instances (members) and checks their states
>
>
> Yes, we need to enhance auto scaler
>
>
>
>       ++ How is dependency retrieved from configuration data (e.g.
> published through topology, beans, etc ….) ?
>
>
> We need to think on more this. IMO, we may need another topic (application
> deployment definition) to retrieved the config data. SM can define this
> based on provide definition, auto scaler and who ever need can get that.
> Topology is maintaining current running state (currently its for a service,
> we need to enhance it to application), so we can used "application
> deployment definition" data to check with current state.
>
>
>
>       ++ Integrate dependency checks into the runtime portion of the
> autoscaler, e.g.  add the checks to the monitor functionality of a
> ClusterMonitor with respective actions (spawning  / terminating instances)
>
>
>
>
> I think we can coupled auto scaling policy with individual
> cartridges/services.
>
>
>
> What do you think ?
>
>
>
>
> We may need to think about deployment policy, I think with the application
> deployment definition (with some coupling ,like some cartridges may need to
> spin up same partition..etc). May be we are redefining deployment policy
> with application deployment definition. (we may obsolete deployment policy)
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
> Sent: Wednesday, April 30, 2014 4:30 AM
>
>
> To: dev@stratos.incubator.apache.org
> Cc: Shaheedur Haque (shahhaqu)
>
> Subject: Re: [Discuss] Grouping of services (cartridges)
>
>
>
> Hi Martin,
>
>
>
> These use cases are very valid, and we should integrate them into 4.1.0.
> Will go through in detail and see how we can incorporate into Stratos.
>
> I have one question, why do we need hierarchical grouping, (I mean group
> inside a group) and use case? Can't we have flat?
>
>
>
> On Wed, Apr 30, 2014 at 9:56 AM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi Lakmal
>
>
>
> Below is a further enhanced proposal to add grouping to apache stratos:
>
>
>
> In a nutshell:
>
>
>
> By grouping cartridges together we can define characteristics which apply
> to all cartridges alike most importantly it will allow us also to define
> dependencies between cartridges. Groups should also be flexible enough to
> not only contain single cartridges but also groups, allowing a hierarchical
> structure.
>
>
>
> Dependencies are defined at the group level, describing the dependency
> relationship of immediate children.
>
> Other group specific properties could be defined which will apply to all
> immediate group members.
>
>
>
> The subscription model will be extended to subscribe to a group in
> addition to a cartridge (either or). To generalize we added the concept of
> a Subscribable which can be either a group or cartridge.
>
>
>
> Since a Subscribale (or more specifically a group) doesn’t have scaling
> characteristics, it seemed appropriate to add the concept of Scalable,
> describing the scalable nature of a cartridge.
>
>
>
> Below is a more detailed description and a corresponding diagram:
>
>
>
> “Class diagram” of the logical model showing the existing description
> files in the Diadem model, and the proposed changes.
>
>
>
> Description: see attachment Slide1.png
>
>
>
> Some notes on the new dependency system:
>
>
>
>   * A Group can be subscribed or a Scalable can be subscribed, whereas
>
>     today, a Cartridge is subscribed.
>
>       o The new Scalable entity is needed because currently
>
>         Subscriptions own autoscale policies, but that doesn’t work when
>
>         a Group contains more than one Cartridge, because we want one
>
>         autoscale policy per Cartridge. However we can’t couple the
>
>         autoscale policy to the Cartridge because it may be re-used in
>
>         other Subscriptions where a different autoscale policy is desired
>
>   * children is the set of Subscribables in the Group.
>
>       o This supports hierarchical Groups.
>
>   * collocate says “these Children must be physically next to each other”
>
>   * startupOrder is a Set of asymmetric pairs (A, B) where A points to B
>
>     but B doesn’t point to A.
>
>       o These pairs define a DAG, which we use to perform a topological
>
>         sort of the children and get a /partial/ ordering.
>
>       o Any member of “children” not in a pair is a singleton started in
>
>         parallel to everything else.
>
>   * The killBehaviour says that when node X dies, we must kill:
>
>       o Everything else (for applications where the loss of any member
>
>         of “children” cannot be recovered with less impact
>
>       o Nothing else (for applications where any element can be restarted)
>
>       o Or just the things “upstream” in the startup order (for
>
>         conventional tiered applications)
>
>
>
> We think this describes all the use-cases we’ll ever see with the
> exception of things like “n instances of A depends on B”; that can be
> considered in the future as an enhancement.
>
>
>
> Startup use cases considered
>
>
>
>   * All start in parallel: don’t specify any pairs => everything is
>
>     equivalent => all start in parallel
>
>   * All start one-after-another (total order): specify a set of pairs
>
>     such that there is a single contiguous list covering all the nodes,
>
>     e.g. A -> B -> C -> D -> E
>
>   * Some start-up dependencies: specify as necessary e.g. [[MiddleTier
>
>     -> Database], [Logging]]
>
>
>
> Kill use cases considered
>
>
>
>   * One dies => all die: set killBehaviour to KillAll
>
>   * One dies => nothing happens: set killBehaviour to killNone
>
>   * One dies => its dependancies die (aka “restart from here”): specify
>
>     a startupOrder DAG and killBehaviour=StartupOrder. E.g. [[MiddleTier
>
>     -> Database], [Logging]], Database dies => MiddleTier is restarted,
>
>     Logging untouched.
>
>
>
> Example instantiation
>
>
>
> Description: see attachment Slide2.png
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
>
>
> From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
> Sent: Sunday, March 30, 2014 9:09 PM
>
>
> To: dev@stratos.incubator.apache.org
> Subject: Re: [Discuss] Grouping of services (cartridges)
>
>
>
>
>
>
>
> On Mon, Mar 31, 2014 at 7:47 AM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> I totally agree, I am also not in favor of complicating existing
> components (e.g. autoscaler).
>
>
>
> However, I am not sure what the alternative might be to solve the
> requirements (e.g. item 1) mentioned below.
>
> The suggestion I made to enhance the autoscaler / rules are based on my
> understanding that the autoscaler already handles similar actions (e.g. VM
> startup, scaling and termination).
>
> It seems to me a logical extension to add additional knowledge to handle
> the boot sequencing, dependent scaling and dependent termination to the
> autoscaler defined by optional properties in the autoscaler policy.
>
>
>
> From my current point of view, CAMP seems a fairly complex specification
> which might require quite some changes to adopt.
>
>
>
> Sorry I could not go through CAMP in detail. Should spend sometime.
>
>
>
>
>
> I do agree that alternatives might exist and should be discussed !?
> Alternatives ?
>
>
>
> +1 for alternatives, we should not take CAMP as it is, we should see how
> we can match with existing Stratos workflow IMO.
>
>
>
>
>
> Thanks
>
>
>
> Martin
>
>
>
> From: damitha kumarage [mailto:damitha23@gmail.com]
> Sent: Sunday, March 30, 2014 7:59 AM
> To: dev@stratos.incubator.apache.org
>
>
> Subject: Re: [Discuss] Grouping of services (cartridges)
>
>
>
> Please see my inline comment,
>
>
>
> On Sat, Mar 29, 2014 at 11:53 AM, Isuru Haththotuwa <isuruh@wso2.com>
> wrote:
>
> Hi Martin,
>
> On Fri, Mar 28, 2014 at 11:19 PM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
> Hi,
>
>
>
> I think this property will be a quite useful piece to solve the grouping
> issue:
>
>
>
> I also would like to suggest to add the serviceGroup to the topology map
> (in case it is not yet available in the topology map). This  will help to
> tie together cartridges (or services) in the autoscaler and , for example
> enable synchronized auto scaling behavior of services within a service
> group, like synchronized scaling, sequenced boot up, etc ….
>
>
>
> In addition the autoscaler should be enhanced to add additional (but
> optional properties) in the auto scaling policy related to a service group
> to govern the respective auto scaling behavior.
>
>
>
> For example, related properties should identify a service group and other
> related properties to define dependencies between the various cartridges in
> the service group like boot sequence, scale up / down ratios, termination
> dependencies, etc … . The property set (or json structure ) should be
> fairly flexible as we are just about to explore this new feature and should
> be easily expandable.
>
>
>
> I would think that these additions will also prove useful to integrate in
> the long term with CAMP (or other spec) but will help to solve more
> immediate requirements
>
>
>
> Yes, these are very valid points in coming up with a proper grouping
> architecture for services. Thank you for bringing them up.
>
> As I understand, what Sajith has done here is enabling static grouping of
> services, by using a property for that in the cartridge deployment time.
> What we are trying to achieve in long term is dynamic grouping of services,
> so that we can group any available service at runtime seamlessly, according
> to CAMP specification (or some other suitable way).
>
>
> I see three things here
> 1) Grouping and resolve inter-dependancies between  cartridges.
>
> 2) Composite Artifact deployment (May be if required, according to the
> above grouping and inter-dependancy( of cartridges. There can also be
> intra-dependancies  inside a cartridge in case of artifact deployment)
>
> 3) Improved nonitoring and health check of above intricacies.
>
> Instead of adopt CAMP to solve above, I think it is better to discuss and
> find ways that fits most naturally to the existing Stratos
> architecture(Unless CAMP is widely adopted and we are compelled to
> adhere).  Is there a way we can solve this without doing major changes to
> existing components? For example without complicating autoscaler
> policies/logic as suggested by Martin above?
>
> Damitha
>
>
>
>
>
>
>
> --
> __________________________________________________________________
> Damitha Kumarage
> http://people.apache.org/
> __________________________________________________________________
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Software Architect; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
>


-- 
Lakmal Warusawithana
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/

Mime
View raw message