Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of MFerreira@schubergphilis.com
 designates 195.66.90.56 as permitted sender)
From: Miguel Ferreira <MFerreira@schubergphilis.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
CC: Alex Huang <Alex.Huang@citrix.com>
Subject: RE: [DISCUSS] Enabling databse upgrades on master branch
Thread-Topic: [DISCUSS] Enabling databse upgrades on master branch
Thread-Index: Ac88cZZ6z8A0LXYpRtiSzGByOxV9lAAPSEoAABgL4yA=
Date: Tue, 11 Mar 2014 09:58:29 +0000
Message-ID: <02E3616756CDC04687D08374DA63D32C4CF000@SBPOMB101.sbp.lan>
References: <02E3616756CDC04687D08374DA63D32C4CD17A@SBPOMB101.sbp.lan>
 <7AB883FD-1FDA-4219-B5C7-C168313D64A8@citrix.com>
In-Reply-To: <7AB883FD-1FDA-4219-B5C7-C168313D64A8@citrix.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Rajani,

Indeed I see the overlap with the previous discussion that is taking place.=
 I actually tried to chip-in that discussion.
I do agree with you on introducing tools that can support the database upgr=
ade process, but meanwhile a small change in the way developers maintain th=
e database related code could already bring us very far.

As of now I'm working on a separate project with the following roadmap:
- detect potential conflicts (and collect data about them)
- learn from the data collected which conflicts can we resolve automaticall=
y, which we cannot
- for the conflicts that we cannot resolve automatically, provide some guid=
ance to the operator on how proceed with the upgrade

I'm working on a repository that is not accessible from the outside of Schu=
berg Philis, but the intention is to move it to github.com ASAP.

I'm very interested in aligning efforts with the rest of the community, and=
 I'm available to help out in whatever is decided. Meanwhile, I will contin=
ue to develop the ideas we have, and anyone interested in helping out with =
that is very welcome.

If you have any ideas on how to collaborate, please let me know.

Cheers,
Miguel

-----Original Message-----
From: Rajani Karuturi [mailto:Rajani.Karuturi@citrix.com]=20
Sent: dinsdag 11 maart 2014 7:19
To: dev@cloudstack.apache.org
Cc: Alex Huang
Subject: Re: [DISCUSS] Enabling databse upgrades on master branch

Hi Miguel,

This is in-line with discussions related to db changes we are having at [1]=
 and [2]

I think it would be better to use existing tools like [liquibase] or [flywa=
y] instead of writing a new one. A good comparison of the both is available=
 at [3].

Also, how do we join the efforts? Is there any design doc? Are you working =
on a separate branch or as separate project?


[1] http://markmail.org/message/r7wv36o356nolq7f
[2] http://markmail.org/message/wlua3l4o5shayidf
[liquibase] http://www.liquibase.org/
[flyway] http://flywaydb.org/
[3] http://stackoverflow.com/a/8489144/201514


~Rajani


On 10-Mar-2014, at 8:35 pm, Miguel Ferreira <MFerreira@schubergphilis.com<m=
ailto:MFerreira@schubergphilis.com>> wrote:

Hi all,

At Schuberg Philis we are interested in upgrading our running installation =
of ACS more frequently than the current release cycle.
To that end, we are working on tooling to detect potentially conflict intro=
ducing changes to the ACS database and upgrade software.
By conflict introducing change I mean a change to ACS that requires a clean=
 database to start with. Thus, rendering a database of a running installati=
on useless.
Once we can detect the changes that introduce conflicts, we will start to m=
onitor them to better understand how to mitigate and possibly work around t=
he conflicts.

One thing we can already foresee as problematic is the way the upgrade soft=
ware (SQL scripts and Java classes) are being maintained. It is part of the=
 current way-of-working to make all kinds of changes to the upgrade softwar=
e in the master branch. Say, if a create statement was introduced in a SQL =
script in commit C1, then the same create statement might be changed in com=
mit C2, or even further down the line. If we want to continuously upgrade o=
ur running installation, we would not be able to upgrade past C1 without at=
 least losing some data. One possible way out of this problem is to add an =
alter statement in C2 instead of changing the create statement introduced i=
n C1.

We are in favor of all changes to the upgrade software being made in such a=
 way that they can be applied independently and incrementally. We do realiz=
e that this entails more effort from developers, but we also see the benefi=
t of enabling continuous delivery: we basically get a shorter feedback loop=
 on the quality of ACS in a real-world scenario.

We are making all tools related to this open source, so anyone that shares =
the same interest is welcome to join the effort.
Lastly, we would like to hear your opinions about the issue I described, as=
 well as, the proposed solution and any other solutions you might come up w=
ith.


Kind regards,
Miguel Ferreira

Mission Critical Engineer
Schuberg Philis
Boeingavenue 271
1119 PD Schiphol-Rijk
schubergphilis.com<http://schubergphilis.com>

+31 207506617
+31 610907106
_____________________