cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leo Simons <LSim...@schubergphilis.com>
Subject Re: rfc: newsystemvm
Date Mon, 21 Jul 2014 14:24:26 GMT
Hey Chiradeep,

Thanks for taking a look. I’ve now re-done this work, but carefully and cleanly and on top
of current master, in 37 small commits instead of 2 scary ones.

Please take a look at

  https://github.com/schubergphilis/cloudstack/compare/systemvm-refactor

Summarizing this kind of thing is always hard...it’s many little things...the interesting
stuff is at the end/bottom, in particular the two main improvements

  https://github.com/schubergphilis/cloudstack/commit/142d087f6a97f6ac70a858a35d2fe8b638c58cbb
	When working on the systemvm in isolation, or using vagrant or similar tools,
	it can be useful to inject a custom SSH key before merging a management server
	systemvm.iso into it. This option allows that. It should _not_ have effect
	on management-server-managed vms which always get their SSH keys injected.

  https://github.com/schubergphilis/cloudstack/commit/e2240eaed18000d4d94dbf6a6e40612db1aeda34
	The current build downloads its script from master by fetching a cloudstack
	tarball. Besides being an unneeded load on the apache git server, this is a
	problem when working on a branch and wanting to inject a different set of
	scripts. It also makes it pretty likely that the injected copy of the script
	will not match what a production release wants, so there is very little
	chance of not needing to overwrite the scripts.

	Ideally we would just rsync over some files. However, veewee does not provide
	an option to do that. In order to keep a 'cleanly veewee-only' build possible,
	and work with any recent veewee version, in this change we restor to using
	shar (http://en.wikipedia.org/wiki/Shar) to produce an archive which can
	execute as a script, which we feed to veewee to execute.

In order to avoid having to re-do this cleanup twice, I also ended up merging the systemvm
and systemvm64 template definitions, factoring out their small differences by inspecting the
os architecture.

  https://github.com/schubergphilis/cloudstack/commit/f570b3921cd52672f841fc5f99cdd96f9737d629
  https://github.com/schubergphilis/cloudstack/commit/50e91217f90fc952182dccac02a5af06ac33fb45

Everything else…well it pretty much falls into two categories:
  * general code cleanup without functional changes
  * general code defensiveness to survive various jenkins build scenarios

All in all it should help with ongoing maintenance, I think.

Note I still have some work to do (testing, merging this version back into our redundant vpc
branch, moar testing, ...) before submitting a merge-able patchset. But since it’s such
a big change and since the testing is a bit slow (what with creating and destroying VMs) any
early comments would be quite useful so I don’t have to re-re-do lots of testing.


Thanks!


Leo

On Jul 18, 2014, at 7:35 PM, Chiradeep Vittal <Chiradeep.Vittal@citrix.com> wrote:

> Thanks Leo. Can you summarize the changes (it is a lot of changes)
> 
> From: Leo Simons <LSimons@schubergphilis.com>
> Reply-To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
> Date: Friday, July 18, 2014 at 7:42 AM
> To: int-toolkit <int-toolkit@schubergphilis.com>, "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
> Subject: rfc: newsystemvm
> 
> Hey folks,
> 
> https://github.com/schubergphilis/cloudstack/commit/f125f1564e8921def00dc0235ecca51470a2a22e
> https://github.com/schubergphilis/cloudstack/tree/f125f1564e8921def00dc0235ecca51470a2a22e/tools/appliance
> 
> This started out as wanting the systemvm build to take systemvm/patches/debian/{debian,vpn}
from the local machine/branch, rather than downloading from the apache git master [1]. In
working out how on earth to get veewee to do that cleanly (hint: you can’t, hence resorting
to shar usage) I got quite frustrated with the image rebuild times.
> 
> It so happens that veewee has a --skip-to-postinstall instruction which is _quite_ useful
while debugging these scripts. To get that working requires the post install steps to be retryable/convergent.
Of course, our existing scripts weren’t set up for that. So I had to add a bunch of tests
whether changes had applied already. Which implied a pretty significant refactor.
> 
> I think I was careful enough and I expect this new template will work just as well as
the old one. This is a change that we can (and probably should?) merge to master independently
of the redundant VPC work (though the `apt-get install chef` would need to be taken out).
But, given how big of a chunk of code has changed here, before upstreaming (a version of)
this to apache we (I) need to do more testing. So for now I’ve put this change next to the
existing definitions rather than replace ‘em, to not block anything else.
> 
> Comments/thoughts?
> 
> 
> cheers,
> 
> 
> Leo
> 
> 
> [1] https://github.com/schubergphilis/cloudstack/blob/master/tools/appliance/definitions/systemvmtemplate/postinstall.sh#L228
> 
> Begin forwarded message:
> ...
>>     M tools/appliance/build.sh
> ...
>>     A tools/appliance/definitions/newsystemvm/apt_upgrade.sh
>>     A tools/appliance/definitions/newsystemvm/authorized_keys.sh
>>     A tools/appliance/definitions/newsystemvm/build_time.sh
>>     A tools/appliance/definitions/newsystemvm/cleanup.sh
>>     A tools/appliance/definitions/newsystemvm/configure_acpid.sh
>>     A tools/appliance/definitions/newsystemvm/configure_conntrack.sh
>>     A tools/appliance/definitions/newsystemvm/configure_grub.sh
>>     A tools/appliance/definitions/newsystemvm/configure_locale.sh
>>     A tools/appliance/definitions/newsystemvm/configure_login.sh
>>     A tools/appliance/definitions/newsystemvm/configure_networking.sh
>>     A tools/appliance/definitions/newsystemvm/configure_systemvm_services.sh
>>     A tools/appliance/definitions/newsystemvm/definition.rb
>>     A tools/appliance/definitions/newsystemvm/install_systemvm_packages.sh
>>     A tools/appliance/definitions/newsystemvm/preseed.cfg
>>     A tools/appliance/definitions/newsystemvm/zerodisk.sh
>>     A tools/appliance/shar_cloud_scripts.sh
>>     A tools/appliance/test.sh
> ...
>> Work in progress to rewrite systemvm box definition in clean code.
>> * Refactor build.sh to use functions and pretty logging.
>> * Add a new test.sh which tries several different build.sh invocations.
>> * Add a new 'debianbase' definition which is just the basic debian box (no systemvm
functionality). This is used for testing.
>> * Add a new 'newsystemvm' definition which is a work-in-progress replacement of the
'systemvmtemplate' definition:
>> ** reduce duplication between preseed.cfg and postinstall.sh
>> ** remove duplication between postinstall.sh and cloud-scripts
>> ** install cloud-scripts from current working copy instead of tying to a download
of the apache master branch
>> ** split up postinstall.sh into small utility scripts and clean up that code
>> ** clean up preseed.cfg to match current veewee/debian best practices and remove
cruft
> 
> 


Mime
View raw message