cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Weingärtner <rafaelweingart...@gmail.com>
Subject Let’s discuss database upgrades
Date Mon, 28 Dec 2015 13:16:59 GMT
Hi all devs,
First of all, sorry the long text, but I hope we can start a discussion
here and improve that part of ACS.

A while ago I have faced the code that Apache CloudStack (ACS) uses to
upgrade from a version to newer one and that did not seem to be a good way
to execute our upgrades. Therefore, I decided to use some time to search
for alternatives.

I have read some material about versioning of scripts used to upgrade a
database (DB) of a system and went through some frameworks that could help
us.

In the literature of software engineering, it is firmly stated that we have
to version DB scripts as we do with the source code of the application,
using the baseline approach. Gladly, we were not that bad at this point, we
already versioned our routines for DB upgrade (.sql and .java). Therefore,
it seemed that we just did not have used a practical approach to help us
during DB upgrades.

>From my readings and looking at the ACS source code I raised the following
requirement:
•    We should be able to write more than one routine to upgrade to a
version; those routines can be written in Java and SQL. We might have more
than a routine to be executed for each version and we should be able to
define an order of execution. Additionally, to go to an upper version, we
have to run all of the routines from smaller versions first, until we
achieve the desired version.

We could also add another requirement that is the downgrade from a version,
which we currently do not support. With that comes my first question for
discussion:
•    Do we want/need a method to downgrade from a version to a previous one?

I found an explanation for not supporting downgrades, and I liked it:
http://flywaydb.org/documentation/faq.html#downgrade

So, what I devised for us:
First the bureaucracy part  - our migrations occur basically in three (3)
steps, first we have a "prepare script", then a cleanup script and finally
the migration per se that is written in Java, at least, that is what we can
expect when reading the interface “com.cloud.upgrade.dao.DbUpgrade”.

Additionally, our scripts have the following naming convention:
schema-<currentVersion>to<desiredVersion>, which in IMHO may cause some
confusion because at first sight we may think that from the same version we
could have different paths to an upper version, which in practice is not
happening. Instead of a <currentVersion>to<version> we could simply use
V_<numberOfVersion>_<sequencial>.<fileExtension>, giving that, we have to
execute all of the V_<version> scripts that are smaller than the version we
want to upgrade.

To clarify what I am saying, I will use an example. Let’s say we have just
installed ACS and ran the cloudstack-setup-database. That command will
create a database schema in version 4.0.0. To upgrade that schema to
version 4.3.0 (it is just an example, it could be any other version), ACS
will use the following mapping:

_upgradeMap.put("4.0.0", new DbUpgrade[] {new Upgrade40to41(), new
Upgrade410to420(), new Upgrade420to421(), new Upgrade421to430())

After loading the mapping, ACS will execute the scripts defined in each one
of the Upgrade path classes and the migration code per se.

Now, let’s say we change the “.sql” scripts name to the pattern I
mentioned, we would have the following scripts; those are the scripts found
that aim to upgrade to versions between the interval 4.0.0 – 4.3.0
(considering 4.3.0, since that is the goal version):


   - schema-40to410, can be named to:  V_410_A.sql
   - schema-40to410-cleanup, can be named to:  V_410_B.sql
   - schema-410to420, can be named to:  V_420_A.sql
   - schema-410to420-cleanup , can be named to:  V_420_b.sql
   - schema-420to421, can be named to:  V_421_A.sql
   - schema-421to430, can be named to:  V_430_A.sql
   - schema-421to430-cleanup, can be named to:  V_430_B.sql


Additionally, all of the java code would have to follow the same
convention. For instance, we have “com.cloud.upgrade.dao.Upgrade40to41”,
which has some java code to migrate from 4.0.0 to 4.1.0. The idea is to
extract that migration code to a Java class named:  V_410_C.java, giving
that it has to execute the SQL scripts before the java code.

In order to go from a smaller version (4.0.0) to an upper one (4.3.0), we
have to run all of the migration routines from intermediate versions. That
is what we are already doing, but we do all of that manually.

Bottom line, I think we could simple use the convention
V_<numberOfVersion>_<sequencial>.<fileExtension> to name upgrade routines.
That would facilitate us to use a framework to help us with that process.
Additionally, I believe that we should always assume that to go from a
smaller version to a higher one, we should run all of the scripts that
exist between them. What do you guys think of that?

After the bureaucracy, we can discuss tools. If we use that convention to
name migration (upgrade) routines, we can start thinking on tools to
support our migration process. I found two (2) promising ones: Liquibase
and Flywaydb (both seem to be under Apache license, but the first one has
an enterprise version?!). After reading the documentation and some usage
examples I found the flywaydb easier and simpler to use.

What are the options of tools that we can use to help us manage the
database upgrade, without needing to code the upgrade path that you know?

After that, I think we should decide if we should create another
project/component to take care of migrations, or we can just  add the
dependency of the tool to a project such as “cloud-framework-db” and start
using it.

The “cloud-framework-db” project seems to have a focus on other things such
as managing transactions and generating SQLs from annotations (?!? That
should be a topic for another discussion). Therefore, I would rather create
a new project that has the specific goal of managing ACS DB upgrades.  I
would also move all of the routines (SQL and Java) to this new project.
This project would be a module of the CloudStack project and it would
execute the upgrade routines at the startup of ACS.

I believe that going from a homemade solution to one that is more
consolidated and used by other communities would be the way to go.

I can volunteer myself to create a PR with the aforementioned changes and
using flywaydb to manage our upgrades. However, I prefer to have a good
discussion with other devs first, before starting coding.

Do you have suggestions or points that should be raised before we start
working on that?

--
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message