incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Schweikert <>
Subject Re: [DISCUSS] Packaging going forward
Date Wed, 21 Nov 2012 19:38:39 GMT
On 11/20/2012 12:18 PM, Hugo Trippaers wrote:
> Hey all,
> Packaging is a work in progress at the moment. Wido, me and others are working on this,
but we are not there yet. Partially this is because there are a multitude of things to consider
when we discuss packaging. Hence this mail to share a lot of the thoughts that went into packaging.
> First of all is how we look at the packages. With the ASF it is pretty clear that the
"release" is the source code. We tag a particular state of the source tree and voila, we have
a release. So "packaging" our "release" is a simple as making at tarball or zip of the code
and making it available to users (aside from the ASF process to name something a release).
> Compiling the code, as part of our build procedure generates artifacts (using the mvn
package target). Artifacts in this sense are jar, war and zip files containing a mix of compiled
java classes, scripts, documentation (and optionally dependencies). Most of these artifacts
are already automagically sent to the apache snapshot repositories by jenkins because we inherit
the maven configuration from the apache master (or as a final release if we wanted to).
> Finally "packaging" is taking the artifacts generated by the compile/build step and turning
it into some kind of OS specific package like an RPM or DEB.
> The current build system based around maven is designed for this way of working. Maven
is only taking responsibility for turning the source code into artifacts and additional scripts
can take those artifacts and combine them into packages. This is explicitly done this way
to not clutter a generic and multiplatform java build process with very os specific packaging
steps. The packaging step should be something like, download sources as tarball, extract,
run mvn package (with any additional profiles you need), generate package from the artifacts.
In the packaging directory there is a centos63 spec file which does this (see %build and %install
> Currently we have released the 4.0.0 version as a source release and some members have
graciously offered to compile, build and package the source and make that available to the
community for their convenience. I think the explicit wish is that we are able to provide
a distribution ourselves (meaning the Apache CloudStack community) instead of just providing
source code. This brings us back to the original discussion regarding licensing and how to
deal with dependencies in packages.
> Wido and myself had a lengthy discussion on this subject at ApacheConEU and we haven't
reached a conclusion yet, but a good enough understanding of the problem that we can bring
our ideas and discussions to the list.
> One of the main arguments is how to deal with dependencies in packages. There are a couple
of options here:
> *         Create self-contained artifacts in step 2 (compile/build) that contain all
required dependencies. Packages made from these artifacts need no outside dependencies save
the bare essentials like java runtime, tomcat and python.
> *         Create "bare" artifacts with just our compiled code in step 2 (compile/build).
Packages need to include all dependencies for all jars that are generated as part of our code.
> *         Hybrid mix and match of the two options above.
> *         The old waf build is even worse. We compile bare artifacts in step 2 (build/compile)
and then package downloaded dependencies as far as the ASF permits and let the package dependencies
deal with the others (like mysql-connector-java).
> In my view one of the big issues here is version management of our dependencies.

Managing dependencies can be conceived as a two pronged effort. However, 
it can also be boiled down to letting packagers deal with it and thus it 
can be externalized/delegated.

One can look at dependency management from a developers or users point 
of view and can decide that the project should provide, in some way, the 
dependencies to potential developers or users. This is reasonable, 
however it leads to the point where packagers for distributions manage 
dependencies and the project manages dependencies. Thus, effort is 
duplicated across the open source community and people's time is wasted.

Considering that developers and users run on some distribution it would 
be a more efficient approach to let packagers worry about the 
dependencies. Then a developer that wants to join the effort can just 
get all the packages from the distro of her/his choice and clone the git 
tree and voila everything is just there and works (famous last words, I 
know ;) ). Users are most likely to try and find the packages in the 
repositories of their distro of choice first before they will go looking 
for packages from the project. Thus if we can get the word out and make 
it easy to package CloudStack we will get good support in the 
distributions and should not have to worry about having packages from 
the project. This way the license concern issue is also avoided.

Therefore, IMHO the project should not really worry too much about 
dependencies, other than clearly documenting them.

The goal should be to provide a build setup that:
- does not download anything
- builds the artifacts
- places the artifacts in a reasonably nice directory structure
- provide a generic guide for packagers that explains what artifacts are 
part of the various CloudStack components

> If we want to ship packages for certain operating systems we need to align our dependency
versions to exactly the versions shipped by those distributions.

And that will be a nightmare for the project. It is much better to leave 
that rat hole alone and let people that contribute to the project worry 
about the core

- how to build it
- how do the various components work together
- cloud management functionality
- documentation

That said, my question is, is the 4.0 release there?

- will maven produce all the artifacts needed or is there still some 
ugly mix of waf, maven, or whatever?
- is there documentation about how the components are supposed to fit 
- is there a document that describes all the dependencies and when they 
come into play. For example in a KVM setup one would not have a 
dependencies on the xen-java stuff.

Unfortunately I have lost track of where things stand, thus please 
excuse me if these questions have been answered or the answers are 
obvious from previous communication during the 4.0 cycle.

Robert Schweikert                           MAY THE SOURCE BE WITH YOU
SUSE-IBM Software Integration Center                   LINUX
Tech Lead

View raw message