Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: <kdamage@apache.org>
Sender: "Kelcey Damage \(BT\)" <kelcey@backbonetechnology.com>
To: <cloudstack-dev@incubator.apache.org>
References: <CD1198E3.4964%min.chen@citrix.com>
 <50EE0DA6.4030305@CloudsoftCorp.com>
 <080901cdef5f$286d0f60$79472e20$@backbonetechnology.com>
 <8BC36449D5CD4B30A18DECA7040DD17B@digitalsanctum.com>
 <CAOvkMoZRYiLjW47bXv3x=JvgDJB5hjwpfOziMVPQVSbhY4K62Q@mail.gmail.com>
 <CAFtUSExa0HvRAOB8TTinormi_AHUV4cOzs6wHdEPSbiVw=wYwQ@mail.gmail.com>
 <FA166DB9-6CEA-415D-B4C1-9051AE11E261@bbits.ca>
 <7A92FF96DF135843B4B608FB576BFC3E012DA320CA31@SJCPMAILBOX01.citrite.net>
In-Reply-To: 
 <7A92FF96DF135843B4B608FB576BFC3E012DA320CA31@SJCPMAILBOX01.citrite.net>
Subject: RE: [DISCUSS] PaaS Enablement: Composite Application Blueprints
 (#576)
Date: Thu, 17 Jan 2013 10:23:32 -0800
Message-ID: <022801cdf4df$c14d85e0$43e891a0$@apache.org>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: 
 AQECOILpaIGFN3ytMaP7zZEhCrOBRgIcKF0LAjhixacCPvtRAwF35lzDAZ63v+4CFoUkKwGKWfA1mXr1jjA=
Content-Language: en-us

My comment about the projects was that as they stand, they are no
recoverable. When you start a project you must expect it to terminate. There
is no easy method to take the work you accomplish in a project and move it
to a production system. 

It would be nice to be able to pack up a whole project in a vApp (Or at
least and 'administrative migration' option, to migrate VM through admin
boundaries)

But I digress as this discussion is not about the projects themselves.

-kd


>-----Original Message-----
>From: Jie Feng [mailto:Jie.Feng@citrix.com]
>Sent: Thursday, January 17, 2013 9:47 AM
>To: cloudstack-dev@incubator.apache.org
>Subject: RE: [DISCUSS] PaaS Enablement: Composite Application Blueprints
>(#576)
>
>Great analysis Chris!
>
>In terms of the blueprint model, I am also in favor of something simple.
>CloudFormation can do what vApp is doing, and more. We only need to
>implement a subset as a starter. For example, allow users to describe
multiple
>VMs, template ID for the VMs, and default service offering/disk offering,
>networking for the interested zones, and startup order. Agree with Kelcey
>that it does not need to be self-contained. We can support
>importing/exporting of the blueprint to vApp (OVA/OVF) which contains both
>the OVF descriptor and the template image files as an add-on feature.
>
>Do we need to map blueprints to project?  I thought project is more like a
>shared workspace. IMO, we should have the flexibility to deploy multiple
>blueprints and any VMs to a project.
>
>Jie
>
>> -----Original Message-----
>> From: Kelceydamage@bbits [mailto:kelcey@bbits.ca]
>> Sent: Thursday, January 17, 2013 1:51 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: [DISCUSS] PaaS Enablement: Composite Application
>> Blueprints
>> (#576)
>>
>> This was a good read, I think the simpler vApp method makes sense for
>> an infrastructure tool. And could be a great opportunity to rework
>> projects into a more useful system.
>>
>> I would not want to make the blueprints too complex, but my main issue
>> with vApp methodology is how a vApp is completely self-contained. They
>> eat up storage like mad, when they should really just be a collection
>> of existing provisioning resources and templates.
>>
>> Example:
>>
>> A vApp when created copies the included VM temples to create a larger
>> template. It would be better for our blueprints to simply point to the
>> included templates like 'symlinks'. Providing almost zero increase to
>> storage(only meta-data) and making the feature seriously competitive.
>>
>> The main logistics issue would be ensuring a blueprint health check
>> incase someone deletes the linked templates underneath. Or maybe a
>> dependancy error when someone tries to remove a template that's linked
>> to one or more blueprints. This error could inform the admin that the
>> template is in use by a blueprint that must be deleted first.
>>
>> Thanks
>>
>> Sent from my iPhone
>>
>> On Jan 16, 2013, at 11:51 PM, Chris Sears <chris.x.sears@sungard.com>
>> wrote:
>>
>> > After reading up a bit more on AWS CloudFormation [1] and
>> > OpenStack's work-alike Heat [2], they actually differ quite a bit
>> > from VMware vApps. I thought it might be worth talking about some of
>the differences.
>> >
>> > CloudFormation is a general purpose provisioning system for AWS
>> > services, but under the hood, it's essentially a workflow engine
>> > (similar to vCenter Orchestrator or HP Operations Orchestrator). The
>> > problem CloudFormation is solving is runbook automation, where you
>> > might have a 20 step process for deploying an app into AWS that you
>> > want to completely automate. It competes with tools like Puppet and
>> > Chef
>> (although they can work together).
>> >
>> > The templates used by CloudFormation are less config files and more
>> > like scripts or XSLT. They include variables, user inputs, outputs,
>> > flow control, and a small standard library of functions. [3]
>> > Templates require testing and debugging. It's possible to have
>> > run-time bugs related to a wait condition that cause a deadlock and
trigger
>a rollback.
>> >
>> > When you execute a CloudFormation template, the resulting set of
>> > provisioned resources is called a "stack". A stack could be a
>> > multi-tier app with several VMs, but it could also be 3 empty VPCs,
>> > an SNS queue, and an IAM user. If we follow a similar convention, a
>> > stack would not necessarily map onto a project. A stack might
>> > include VMs in multiple projects. Or no VMs at all.
>> >
>> > To extend the workflow coordination and configuration capabilities
>> > into the guest OS, CloudFormation provides a set of helper scripts
>> > that can fill a similar roll as cloud-init, creating files and
>> > running commands with content and arguments dynamically generated
>> > from the template. [4]
>> >
>> > It helped me to look at some real templates to understand everything
>> > that CloudFormation was doing. Here are a couple examples... (the
>> > first provisions an Active Directory server, the second creates an
>> > instance of
>> > OpenShift)
>> > https://s3.amazonaws.com/cloudformation-templates-us-east-
>> 1/Windows_Si
>> > ngle_Server_Active_Directory.template
>> >
>>
>https://github.com/openstack/heat/blob/master/templates/OpenShift.tem
>> p
>> > late
>> >
>> > By contrast, vApps are much simpler. They are effectively just a
>> > collection of one or more VMs, their associated network settings and
>> > some limited additional metadata. You can start and stop a vApp. You
>> > can import or export a vApp as an OVF file if you wanted to move it
>> > from one cloud to another. A vApp could roughly map to a project.
>> >
>> > To me, it sounds like we need vApps, with some limited features of
>> > CloudFormation. Just enough to enable a user experience of clicking
>> > a "Launch Blueprint" button and having everything magically
>> > provisioned, like AWS does here:
>> > http://aws.amazon.com/cloudformation/aws-cloudformation-templates/
>> >
>> > Thoughts?
>> >
>> > - Chris
>> >
>> > [1]: http://aws.amazon.com/cloudformation/faqs/
>> > [2]: https://github.com/openstack/heat
>> > [3]:
>> >
>>
>http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/templ
>> ate
>> > -anatomy.html
>> > [4]:
>> >
>http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-
>> help
>> > er-scripts-reference.html
>> >
>> >
>> > On Wed, Jan 16, 2013 at 1:59 PM, Mohammad Nour El-Din <
>> > nour.mohammad@gmail.com> wrote:
>> >
>> >> Hi
>> >>
>> >> Sent from my Samsung Galaxy S3
>> >> Apologies for any typos
>> >> On Jan 16, 2013 6:01 PM, "Shane Witbeck" <shane@digitalsanctum.com>
>> wrote:
>> >>>
>> >>> +1 for YAML. After all, YAML is a superset of JSON [1] which is
>> >>> +the
>> >> primary objective of the 1.2 spec.
>> >>>
>> >>> Here are my pros and cons:
>> >>>
>> >>> Pros:
>> >>> - relational anchors and aliases (potential here for less
>> >>> verbosity
>> >> through reuse)
>> >>> - more readable
>> >>> - comments
>> >>> - projects which I respect, choose yaml over json for similar
>> >>> types of
>> >> configuration [2], [3]
>> >>>
>> >>> Cons:
>> >>> - more complex to parse, less mature libraries compared to json
>> >>>
>> >>> All this being said, I would advocate that more initial focus
>> >>> should be
>> >> on the "model" and how it solves the problem of defining the
>"blueprint".
>> >> Serialization format discussion should be secondary and around
>> >> whatever best fits the model, not the other way around.
>> >>
>> >> +1 on that
>> >>
>> >>>
>> >>>
>> >>> [1] http://yaml.org/spec/1.2/spec.html#id2759572
>> >>> [2]
>> >> http://www.elasticsearch.org/guide/reference/setup/configuration.ht
>> >> ml
>> >>> [3]
>> >> https://github.com/cloudfoundry/bosh-
>> release/blob/master/micro/aws.ym
>> >> l
>> >>>
>> >>>
>> >>> --
>> >>> Shane Witbeck
>> >>>
>> >>>
>> >>> On Thursday, January 10, 2013 at 1:20 PM, Kelcey Damage (BT) wrote:
>> >>>
>> >>>> +1 for YAML, I find JSON good for when I never look at the data
>> >>>> +and
>> >> simply pass to a Python dictionary, however for interaction I am
>> >> going to back the YAML choice.
>> >>>>
>> >>>> KELCEY DAMAGE
>> >>>> Infrastructure Systems Architect
>> >>>> www.backbonetechnology.com
>> (http://www.backbonetechnology.com)
>> >>>>
>> >> -------------------------------------------------------------------
>> >> --
>> >> ----
>> >>>> kelcey@backbonetechnology.com
>> >>>> (mailto:kelcey@backbonetechnology.com)
>> >>>>
>> >>>> address: 55 East 7th Ave, Vancouver, BC, V5T 1M4
>> >>>> tel: +1 604 713 8560 ext:114
>> >>>> fax: +1 604 605 0964
>> >>>> skype: kelcey.damage
>> >>>>
>> >>>>
>> >>>>
>> >>>>> -----Original Message-----
>> >>>>> From: Alex Heneveld [mailto:alex.heneveld@cloudsoftcorp.com]
>> >>>>> Sent: Wednesday, January 09, 2013 4:39 PM
>> >>>>> To: cloudstack-dev@incubator.apache.org (mailto:
>> >> cloudstack-dev@incubator.apache.org)
>> >>>>> Cc: Min Chen
>> >>>>> Subject: Re: [DISCUSS] PaaS Enablement: Composite Application
>> >> Blueprints
>> >>>>> (#576)
>> >>>>>
>> >>>>>
>> >>>>> Hi Min, Jie,
>> >>>>>
>> >>>>> Min, nice questions. I've given my answers in-line.
>> >>>>>
>> >>>>> Jie, my answers also respond to some of your points, re TOSCA
>> >>>>> and YAML/JSON.
>> >>>>>
>> >>>>>
>> >>>>> On 08/01/2013 17:50, Min Chen wrote:
>> >>>>>> +1 on this feature to extend CloudStack from pure IaaS to PaaS
>> >>>>>> +or
>> >> SaaS
>> >>>>>> area. Some questions in my mind:
>> >>>>>>
>> >>>>>> 1) Does your proposed blueprint only contain functional
>> >>>>>> component description or contain both functional and deployment
>aspects?
>> >>>>>> For example, for a 3-tier app containing three functional
>> >>>>>> components
>> >> like
>> >>>>>> web tier, server, and DB, users can define different deployment
>> >> models
>> >>>>>> for the same functional components, like small deployment
>> >> (co-hosting
>> >>>>>> all components on one VM), or medium or large deployment.
>> >>>>>>
>> >>>>>
>> >>>>> Great use case. Introduces a _lot_ of complexity:
>> >>>>>
>> >>>>> (1) allow parameters to be passed to the blueprint (e.g. "size")
>> >>>>> (2) separate the components that are to be created from the
>> >> customizations
>> >>>>> performed
>> >>>>> (so you could define customizations for webserver, appserver,
>> >>>>> and
>> >> data,
>> >>>>> then either
>> >>>>> apply the customizations either to a single VM (small), distinct
>> >>>>> VM's (medium), or distinct VM groups (large))
>> >>>>> (3) allow blueprints to refer to other blueprints (so the
>> >>>>> "webserver customization" lives as its own reusable module)
>> >>>>> (4) allow conditional branching based on parameters (e.g. if
>> >>>>> user selects "size=small" then run all customisations on
>> >> single VM)
>> >>>>>
>> >>>>> I'd be inclined to DEFER them as features, until phase 2 or 3,
>> >>>>> but we
>> >> should
>> >>>>> consider how the description might support them as I'd like to
>> >>>>> see
>> >> some of
>> >>>>> this functionality eventually.
>> >>>>>
>> >>>>> What do other people think? This is obvious functionality to
>> >>>>> want,
>> >> what's the
>> >>>>> right way to support it but without making the description
>> >>>>> language complicated?
>> >>>>>
>> >>>>> (BTW one of the reasons I think TOSCA could be a good choice is
>> >>>>> that
>> >> it has
>> >>>>> considered many of these. You can define properties (1 above),
>> >> arbitrary
>> >>>>> nodes (2), and references to other definitions (3). It also has
>> >>>>> the
>> >> concepts of
>> >>>>> requirements / capabilities, relationships, policies, and plans
>> >>>>> --
>> >> some of which
>> >>>>> could be used to support (4)
>> >>>>> eventually.)
>> >>>>>
>> >>>>>> 2) Have you thought of creating Service offerings from your
>> >>>>>> defined blueprints? This may allow CloudStack to provide some
>> >>>>>> SaaS
>> >> functionalities.
>> >>>>>>
>> >>>>>
>> >>>>> I like this idea, although it might break assumptions made
>> >>>>> elsewhere
>> >> about
>> >>>>> service offerings so we should look at it carefully.
>> >>>>>
>> >>>>>> 3) As for blueprint description language, easy-to-read and
>> >>>>>> easy-to-edit by human being should be necessary if we don't
>> >>>>>> provide any visual GUI tools to create/edit a blueprint. I have
>> >>>>>> seen some
>> >> JSON
>> >>>>>> format before in BMC Cloud LifeCycle product 2.0.
>> >>>>>>
>> >>>>>
>> >>>>> Wow JSON seems popular so far. And yet YAML is more concise and
>> >>>>> expressive. Designed to be easy for people to read and write
>> >> configuration
>> >>>>> just like this. Whereas JSON is designed for serializing
>> >>>>> objects, in
>> >> a way that
>> >>>>> isn't too hard for people. YAML seems the right language for
>> >>>>> this
>> >> purpose to
>> >>>>> me -- and my experience has been that it is easier to read and
>> >>>>> to
>> >> write,
>> >>>>> without the litter of curly braces and quotes.
>> >>>>>
>> >>>>> But I don't want to be a language bigot! I can go with JSON if
>> >>>>> that's
>> >> what
>> >>>>> people want. :)
>> >>>>>
>> >>>>>> 4) For a multi-tier application, blueprint should not only
>> >>>>>> describe different components, but also connections among those
>> >>>>>> components, like open port, LB and FW rules among them. I
>> >>>>>> assume that your blueprint is also taking that into
>> >>>>>> consideration, and your backend orchestration layer can provide
>> >>>>>> enough flexibility to provision any such app, seems not an easy
>> >>>>>> task to me since I am
>> new to CS.
>> >>>>>>
>> >>>>>
>> >>>>> Yeah, this is where it gets interesting. A common trick I've
>> >>>>> seen is
>> >> to use
>> >>>>> hostnames for a lot of the connection configuration, and have a
>> >>>>> few
>> >> hard-
>> >>>>> coded patterns in there (e.g. for groups, load-balancers). But
>> >>>>> that
>> >> only gets
>> >>>>> you so far, and then you hit a wall (or spin up a different
>> >> orchestrator).
>> >>>>>
>> >>>>> Using requirements/capabilities and typed relationships gives a
>> >>>>> much
>> >> more an
>> >>>>> elegant solution. (Not necessarily TOSCA, but these concepts are
>> >>>>> well developed there!)
>> >>>>>
>> >>>>> In any case let's make sure the use cases include interesting
>> >> situations like
>> >>>>> this, and we can compare approaches. This absolutely DOES need
>> >>>>> to be
>> >> in
>> >>>>> phase 1 I think.
>> >>>>>
>> >>>>>> 5) From a normal user perspective, I would really love to see a
>> >>>>>> GUI tool developed around this to allow user to visually
>> >>>>>> create/edit a multi-tier app blueprint instead of using a
TextEditor.
>> >>>>>>
>> >>>>>
>> >>>>> Nice to hear. Me too. The description, API, and backend come
>> >>>>> first,
>> >> with an
>> >>>>> initial set of features, but I'd love to see the GUI in phase 2.
>> >>>>>
>> >>>>>> Thanks
>> >>>>>> -min
>> >>>>>>
>> >>>>>
>> >>>>> Keep it coming please.
>> >>>>>
>> >>>>> --Alex
>> >>>
>> >>