brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aled Sage <aled.s...@gmail.com>
Subject Re: YAML blueprint: install package utility
Date Tue, 24 May 2016 09:37:50 GMT
Hi all,

TL;DR: how about the following, where there is a "handler" registered to 
process the brooklyn.packages section?

    - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
       brooklyn.packages:
         yum: curl
         apt: curl-whatever=2.3.4

Or is this over engineering?!

---
To me, the "brooklyn.initializers" and the 
"org.apache.brooklyn.initializer.stock.PackageInstaller" are 
implementation details that a YAML blueprint author would never want to 
type.

I also agree that we don't want "packages" as a config key: as Duncan 
Grant described, that moves VanillaSoftwareProcess towards being a "god 
object" with more and more config on that class.

---
I think there is an alternative where we add first-class support for 
more things in YAML. For example, we should get rid of the need for 
"brooklyn.initializers" when defining sensors, effectors and feeds - we 
should be able to have a section "brooklyn.sensors:".

The same could apply for packages.

However, "packages" are not a first-class concept in Brooklyn (unlike 
sensors, effectors, etc).

---
To take a step back...

The use of "brooklyn.initializers" is to declare mixins [1]. The entries 
under brooklyn.initializers are the types + config of each mixin. This 
gives a very direct translation from YAML to the underlying Java api.

Instead, one could register a handler for a given YAML key (of the form 
"brooklyn.xyz"). This would apply only to top-level entries against an 
entity, policy, enricher, etc (e.g. not when embedded inside a 
brooklyn.config section). Under-the-covers, the handler would return 
something of type EntityInitializer (or equivalent for 
policy/enricher/location). Or perhaps it could even act directly on the 
EntitySpec that is being created.

---
When implementing this, we'd separate it into a number of parts:

  * Implement Andrew's
    org.apache.brooklyn.initializer.stock.PackageInstaller
  * Add support for registering yaml "handlers"
  * Register a yaml handler for "brooklyn.packages", which instantiates
    Andrew's PackageInstaller.

---
I also worry that for installing packages, things are often more 
complicated than these simple use-cases. For example, when installing 
Docker on CentOS7, you may first need to populate the file 
/etc/yum.repos.d/docker.repo [2]. Or you may need more complicated 
install commands, such as first executing yum upgrade ca-certificates 
--disablerepo=epel [3].

The ordering of executing `brooklyn.packages` (compared to other 
commands on VanillaSoftwareProcess) would not be obvious to a blueprint 
author.

For some use-cases, a bash script becomes easier to write and test, 
rather than trying to support all these variations.

However, making simple cases simpler is a good thing.

---
Thoughts?

Aled

[1] https://en.wikipedia.org/wiki/Mixin
[2] https://docs.docker.com/engine/installation/linux/centos/
[3] http://serverfault.com/a/654660/323894


On 24/05/2016 09:57, Thomas Bouron wrote:
> I quite like Andrew's approach using an initializer, feels elegant and
> relatively easy to implement.
>
> Regarding the syntax, getting a rid of the `brooklyn.initializer` and
> `type` is good for a discoverability point of view but that means going
> through a config key which feels wrong in that case. I don't have an
> argument for it, it's more a gut feeling.
>
> On Tue, 24 May 2016 at 09:37 John McCabe <john@johnmccabe.net> wrote:
>
>> Is there any way we could get rid of the initializers/type parts, rather
>> than:
>>
>> - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
>>    brooklyn.initializers:
>>    - type: org.apache.brooklyn.initializer.stock.PackageInstaller
>>      packages:
>>        yum: curl
>>        apt: curl-whatever=2.3.4
>>
>> Something like:
>>
>> - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
>>    packages:
>>      yum: curl
>>      apt: curl-whatever=2.3.4
>>
>>
>> On Tue, 24 May 2016 at 09:34 Duncan Grant <duncan.grant@cloudsoftcorp.com>
>> wrote:
>>
>>> Andrew,
>>>
>>> good point.  That's much neater.  Should be pretty straightforward to
>>> implement as well.
>>>
>>> Duncan
>>>
>>> On Mon, 23 May 2016 at 21:16 Andrew Kennedy <
>>> andrew.kennedy@cloudsoftcorp.com> wrote:
>>>
>>>> Duncan,
>>>>
>>>> I like the idea here, of having some extra object that does the package
>>>> installation, but what about a customizer or initializer instead? You
>>> could
>>>> have something like this:
>>>>
>>>> - type: org.apache.brooklyn.entity.basic.VanillaSoftwareProcess
>>>>    brooklyn.initializers:
>>>>    - type: org.apache.brooklyn.initializer.stock.PackageInstaller
>>>>      packages:
>>>>        yum: curl
>>>>        apt: curl-whatever=2.3.4
>>>>
>>>> I initially thought of a JcloudsLocationCustomizer but that fails when
>> we
>>>> use BYON locations, so I am thinking of something more like a generic
>>>> entity customizer that runs after the entity has been created, and the
>>>> location is available and SSHable. There are lots of other things these
>>>> initializers could do, like setting up users, configuring networking,
>>>> mounting volumes and so on. I don't think its a big stretch to add this
>>>> functionality here.
>>>>
>>>> Andrew.
>>>>
>>>> On Fri, 6 May 2016 at 14:40 Duncan Grant <
>> duncan.grant@cloudsoftcorp.com
>>>> wrote:
>>>>
>>>>> I agree that this is a problem that needs solved but I have
>> misgivings
>>>>> about each of the proposed solutions.
>>>>>
>>>>> Firstly I think that "discovering" how to write brooklyn yaml can be
>>>>> difficult.  It can be difficult to find things in the documentation
>> and
>>>> if
>>>>> you don't know to look for them in the first place then your only
>> hope
>>> is
>>>>> coming across an example.
>>>>>
>>>>> So if we pre-install a set of bash scripts before running the install
>>>>> scripts I don't see any obvious way for someone using brooklyn to
>> find
>>>> out
>>>>> about those scripts, and when the scripts change I imagine people
>> won't
>>>>> notice the new functionality either.
>>>>> Another downside to using scripts is that it will become more
>> difficult
>>>> to
>>>>> debug script failures.  At the moment I can copy the contents of
>> stdin
>>>> and
>>>>> see the error associated.  If I had to step into scripts in other
>> files
>>>> it
>>>>> could become much harder to find a failure.  In my opinion this ease
>> of
>>>>> reproducibility and seeing the code I am running is one of the
>>> advantages
>>>>> of using bash over just using chef/puppet/salt/etc.
>>>>> If we could find a widely used, well tested bash library then some of
>>>> these
>>>>> issues might be mitigated but I've yet to find one.
>>>>>
>>>>> There is a similar issue with discovering the behaviour of the DSL
>>>> option.
>>>>> I think that the DSL only works because we keep it to a minimum and
>>>> really
>>>>> it only does one task - which is run-time configuring relationships
>>>> between
>>>>> entities.  If we extend it then we have replaced java with dsl and I
>>>> don't
>>>>> think that benefits anyone.  One advantage of the DSL option is that
>> it
>>>>> will generate bash so it shouldn't make debugging any harder.
>>>>>
>>>>> The advantage of adding config options to the VanillaSoftwareProcess
>> is
>>>>> that it adds to extra ways of discovering the behaviour - through the
>>>>> composer's autocomplete and in javadoc.  I assume that this would
>>>> generate
>>>>> extra tasks so we could again debug the bash used and it might make
>> it
>>>>> easier as we might be able to see exactly which package failed to
>>>> install.
>>>>> However the big downside to this is that we are starting to make a
>> sort
>>>> of
>>>>> god object which contains all brooklyn functionality in one object -
>>> i.e.
>>>>> VanilllaSoftwareProcess and I'm not sure that's where we want to go.
>>>>>
>>>>> After all that I don't really have a good answer.
>>>>>
>>>>> My not too good answer would be to make more use of the brooklyn
>> child
>>>>> relationship so that a) at least the yaml could be composed from
>> small
>>>>> pieces of code b) we could write a InstallPackage entity to simplify
>>> the
>>>>> yaml.
>>>>>
>>>>> This might look something like:
>>>>>
>>>>> - type: brooklyn.entity.basic.VanillaSoftwareProcess
>>>>>    install.command: curl somefile > somfile.txt
>>>>>    brooklyn.config:
>>>>>      children.startable.mode: foreground_early
>>>>>    brooklyn.children:
>>>>>    - type: brooklyn.entity.basic.VanillaSoftwareProcess
>>>>>      install.command: |
>>>>>          which curl || \
>>>>>              { sudo apt-get update && sudo apt-get install curl
; }
>> || \
>>>>>              { sudo yum update && sudo yum install curl ; } ||
\
>>>>>              { echo WARNING: cannot install curl && exit 1 ;
}
>>>>> or
>>>>>    - type: brooklyn.entity.basic.InstallPackage
>>>>>      package: curl
>>>>> or
>>>>>    - type: brooklyn.entity.basic.InstallPackage
>>>>>      packages:
>>>>>        yum: curl
>>>>>        apt: someOtherCurl
>>>>>
>>>>> This way everything is discoverable through documentation, javadoc,
>> and
>>>>> autocomplete.  Common requirements like curl are re-usable. Generated
>>>> bash
>>>>> is explicit and if curl fails then it will go on fire.
>>>>> On the other hand it requires longer yaml, isn't as readable, and due
>>> to
>>>>> the use of child parent relationships has a bit of a learning hurdle
>>>>>
>>>>> Regards
>>>>>
>>>>> Duncan
>>>>>
>>>>>
>>>>> On Wed, 4 May 2016 at 17:35 Thomas Bouron <
>>>> thomas.bouron@cloudsoftcorp.com
>>>>> wrote:
>>>>>
>>>>>> Hi Guglielmo.
>>>>>>
>>>>>> I'm not sure to follow your idea of "stacks", could you elaborate
>> on
>>>> that
>>>>>> please?
>>>>>>
>>>>>> Best.
>>>>>>
>>>>>> On Tue, 3 May 2016, 13:38 Guglielmo Nigri, <
>>>>>> guglielmo.nigri@cloudsoftcorp.com> wrote:
>>>>>>
>>>>>>> I propose a declarative approach. For example we could add a
>>>> configKey
>>>>> to
>>>>>>> VanillaSoftwareProcess called requiredPackages.
>>>>>>>
>>>>>>> This way, one could just specify the prerequisite packages, sort
>> of
>>>>>>> dependencies for the software process -- this would also open
up
>>>>>>> interesting possibilities such as “stacks” as bundled
>> dependencies.
>>>>>>> A declarative approach would also work for Windows (or non-bash
>>>>>>> environments).
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Guglielmo
>>>>>>>
>>>>>>>
>>>>>>> On 3 May 2016 at 14:28, Andrea Turli <
>>> andrea.turli@cloudsoftcorp.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Great discussion guys!
>>>>>>>>
>>>>>>>> It seems that it is a shared need, and I wanted to discuss
with
>>> you
>>>>> as
>>>>>> I
>>>>>>>> was not sure about my approach.
>>>>>>>>
>>>>>>>> Andrew,
>>>>>>>>
>>>>>>>> I like your proposal, thanks!
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrea
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3 May 2016 at 13:05, Aled Sage <aled.sage@gmail.com>
wrote:
>>>>>>>>
>>>>>>>>> I lean towards Andrew's approach, rather than a special
>>>>>>>>> $brooklyn.installPackage. Note that different distros
use
>>>> different
>>>>>>>> package
>>>>>>>>> names sometimes, so the parameters to such a function
can get
>>>>>> annoying.
>>>>>>>>> I've been hesitant about us going down the road of
>>>>>>> `brooklyn-commands.sh`
>>>>>>>>> for achieving portable blueprints. It feels like we are
>>>> increasing
>>>>>> the
>>>>>>>>> overlap with things like Chef, Salt, Ansible and Puppet
>> (which
>>> we
>>>>> can
>>>>>>>> build
>>>>>>>>> on top of with Brooklyn).
>>>>>>>>>
>>>>>>>>> However, if we keep brooklyn-commands.sh small and focused,
>>> then
>>>>>> maybe
>>>>>>>>> it's a good idea.
>>>>>>>>>
>>>>>>>>> Aled
>>>>>>>>>
>>>>>>>>> p.s. I want to decrease the use of
>>>>>> `BashCommands.installExecutable()`.
>>>>>>>> The
>>>>>>>>> bash command it generates, with all the alternatives
and
>>>>> parentheses,
>>>>>>>> does
>>>>>>>>> not look like nice Bash. A brooklyn-commands.sh could
do it
>>> much
>>>>>>> cleaner,
>>>>>>>>> with a properly structured multi-line if...elif... (or
>>> whatever).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03/05/2016 11:43, Andrew Kennedy wrote:
>>>>>>>>>
>>>>>>>>>> How about an alternative approach - for all SoftwareProcess
>>>>>> entities,
>>>>>>>>>> Brooklyn will copy over a script file called
>>>>> `brooklyn-commands.sh`
>>>>>>> and
>>>>>>>>>> the
>>>>>>>>>> entity commands will soutce this script before running
the
>>> rest
>>>> of
>>>>>>> their
>>>>>>>>>> configured commands. The script will contain OS agnostic
>>>> functions
>>>>>>>> written
>>>>>>>>>> in Bash that do things like install packages, download
Curl,
>>>> get a
>>>>>>> file
>>>>>>>>>> from a URL etc. Then, the install config might look
like
>> this:
>>>>>>>>>> ```
>>>>>>>>>> install.commands: |
>>>>>>>>>>     brooklyn-installPackages apt:openjdk-1.8.0
>>>>>>>> yum:java-1.8.0-openjdk-devel
>>>>>>>>>> java-1.8.0
>>>>>>>>>>     brooklyn-installPackages curl
>>>>>>>>>>     brooklyn-runAsRoot cp /tmp/whatever /etc/hosts
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> Andrew.
>>>>>>>>>>
>>>>>>>>>> On Tue, 3 May 2016 at 11:24 Andrea Turli <
>>>>>>>> andrea.turli@cloudsoftcorp.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> hi,
>>>>>>>>>>> I’ve been thinking about an utility to simplify
the YAML
>>>>> blueprint
>>>>>>>>>>> creation: from my experience when using
>>> VanillaSoftwareProcess
>>>> is
>>>>>>>>>>> annoying
>>>>>>>>>>> to write a portable script just to install a
package (say,
>>>> java)
>>>>>>> valid
>>>>>>>>>>> for
>>>>>>>>>>> apt, yum, etc so I usually write it (multiple
times) just
>> for
>>>> an
>>>>>> OS.
>>>>>>>>>>> To increase the portability of the YAML blueprint
I’d like
>> to
>>>>>> suggest
>>>>>>>> we
>>>>>>>>>>> extend the brooklyn DSL with something like:
>>>>>>>>>>> ```
>>>>>>>>>>> $brooklyn.installPackage(“curl”)
>>>>>>>>>>> $brooklyn:installPackage({"apt", "openjdk-1.8.0",
"yum",
>>>>>>>>>>> "java-1.8.0-openjdk-devel"}, "java-1.8.0")
>>>>>>>>>>> ```
>>>>>>>>>>> instead of things like
>>>>>>>>>>> ```
>>>>>>>>>>> which curl || \
>>>>>>>>>>>           { sudo apt-get update && sudo
apt-get install curl
>>> ; }
>>>>> ||
>>>>>> \
>>>>>>>>>>>           { sudo yum update && sudo yum
install curl ; } ||
>> \
>>>>>>>>>>>           { echo WARNING: cannot install curl
&& exit 1 ; }
>>>>>>>>>>> ```
>>>>>>>>>>> I’m not entirely sure this feature fits well
on the DSL.
>>>>>>>>>>>
>>>>>>>>>>> Alternatively, we could add a configKey to
>>>> VanillaSoftwareProcess
>>>>>>>> called
>>>>>>>>>>> requiredPackages for a more declarative approach
>>> (@googlielmo's
>>>>>> idea)
>>>>>>>>>>> Wdyt?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>> --
>>>>>> Thomas Bouron • Software Engineer @ Cloudsoft Corporation •
>>>>>> http://www.cloudsoftcorp.com/
>>>>>> Github: https://github.com/tbouron
>>>>>> Twitter: https://twitter.com/eltibouron
>>>>>>
>>>> --
>>>>
>>>> Andrew Kennedy ; Founder clocker.io project ; @grkvlt ; Cloudsoft
>>>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message