cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hugo Trippaers <>
Subject [DISCUSS] Packaging going forward
Date Tue, 20 Nov 2012 17:18:38 GMT
Hey all,

Packaging is a work in progress at the moment. Wido, me and others are working on this, but
we are not there yet. Partially this is because there are a multitude of things to consider
when we discuss packaging. Hence this mail to share a lot of the thoughts that went into packaging.

First of all is how we look at the packages. With the ASF it is pretty clear that the "release"
is the source code. We tag a particular state of the source tree and voila, we have a release.
So "packaging" our "release" is a simple as making at tarball or zip of the code and making
it available to users (aside from the ASF process to name something a release).
Compiling the code, as part of our build procedure generates artifacts (using the mvn package
target). Artifacts in this sense are jar, war and zip files containing a mix of compiled java
classes, scripts, documentation (and optionally dependencies). Most of these artifacts are
already automagically sent to the apache snapshot repositories by jenkins because we inherit
the maven configuration from the apache master (or as a final release if we wanted to).
Finally "packaging" is taking the artifacts generated by the compile/build step and turning
it into some kind of OS specific package like an RPM or DEB.

The current build system based around maven is designed for this way of working. Maven is
only taking responsibility for turning the source code into artifacts and additional scripts
can take those artifacts and combine them into packages. This is explicitly done this way
to not clutter a generic and multiplatform java build process with very os specific packaging
steps. The packaging step should be something like, download sources as tarball, extract,
run mvn package (with any additional profiles you need), generate package from the artifacts.
In the packaging directory there is a centos63 spec file which does this (see %build and %install

Currently we have released the 4.0.0 version as a source release and some members have graciously
offered to compile, build and package the source and make that available to the community
for their convenience. I think the explicit wish is that we are able to provide a distribution
ourselves (meaning the Apache CloudStack community) instead of just providing source code.
This brings us back to the original discussion regarding licensing and how to deal with dependencies
in packages.

Wido and myself had a lengthy discussion on this subject at ApacheConEU and we haven't reached
a conclusion yet, but a good enough understanding of the problem that we can bring our ideas
and discussions to the list.

One of the main arguments is how to deal with dependencies in packages. There are a couple
of options here:

*         Create self-contained artifacts in step 2 (compile/build) that contain all required
dependencies. Packages made from these artifacts need no outside dependencies save the bare
essentials like java runtime, tomcat and python.

*         Create "bare" artifacts with just our compiled code in step 2 (compile/build). Packages
need to include all dependencies for all jars that are generated as part of our code.

*         Hybrid mix and match of the two options above.

*         The old waf build is even worse. We compile bare artifacts in step 2 (build/compile)
and then package downloaded dependencies as far as the ASF permits and let the package dependencies
deal with the others (like mysql-connector-java).

In my view one of the big issues here is version management of our dependencies. If we want
to ship packages for certain operating systems we need to align our dependency versions to
exactly the versions shipped by those distributions. I know that usually a higher or lower
version of a java dependency will just work, but history has proven that you should not rely
on this. If we have tested our version of CloudStack with version 5.1.12, should we ship a
package with a dependency on that specific version of the mysql-connector-java, or hope for
the best and allow any version. (leading the witness I know.)  We also have quite some dependencies
on ancient libraries (> 3 years old) that might not be available as packages at all. Some
dependencies might not even have packages at all or at least not in the distributions repo.

Next question is where to put stuff we package. We toss a lot of things in /usr/share/java
for example, it some cases with the cloud- prefix so they don't clash with system packages.
Is this the right place/method to put our dependencies?

My current take on all this is to go for the completely self-contained option, meaning we
package everything including dependencies into our artifacts and completely ignore any system
dependencies. This is motivated partially by my idea that we should leave packaging into OS
specific packages to the distributions anyway, that way we don't have to deal with startup
scripts, configuration scripts and the whole bunch of distribution specific locations. For
me this give the advantage of not having to worry about specific versions of system dependencies
(and version conflicts with other java applications installed via packages) and I don't have
to deal with managing package dependencies in our packaging code. However, as Chip pointed
out today, these packages could not be shipped from ASF infra as some of the binaries have
incompatible licenses (like mysql-connector-java). I know Wido is far more in favor of packaging
only the bare artifacts and leaving dependencies to the distributions as much as possible,
but he agrees with me that version management of specific dependencies is tricky.

I think it's a bridge to far now to deal with the packaging and license issues for binary
packages and their dependencies. I'd rather focus the effort on generating a clear set of
artifacts in a very clear directory structure that packagers can use to generate their packages.
I'm also not against of shipping spec files and the debian build files, it would be great
if somebody could just do "rpmbuild -tb cloudstack-4.1.0.tar.gz' and have a set of packages
after a few minutes, but very limited to just one or two specific versions of distributions
(or even just generate a generic rpm and dep that just installs our files and leaves specific
stuff to the sysadmin)

So what do you guys think?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message