cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@gmail.com>
Subject Re: Let’s discuss database upgrades
Date Tue, 29 Dec 2015 13:48:28 GMT
By this I meant I'm fine with any decision the implementer makes, btw. We
will encounter the administrative issues with it and deal with them.

On Tue, Dec 29, 2015 at 2:46 PM, Daan Hoogland <daan.hoogland@gmail.com>
wrote:

> Wido, Rafael,
>
> I like the date-format but then of course CCYY-MM-DD. I can still think of
> ways to screw up that (or the plain int;)
>
> On Tue, Dec 29, 2015 at 1:40 PM, Rafael Weingärtner <
> rafaelweingartner@gmail.com> wrote:
>
>> Wido, that is true, you are right; the naming on upgrade routines can use
>> a
>> numeric value independent of the number of the version. The numeric value
>> can be a simple integer that is incremented each routine that is added or
>> a
>> time stamp when the routine was added. The point is that we would have to
>> link a version to a number. That would enable us to use flywaydb.
>>
>> To use that approach I think we might need to break compatibility as you
>> pointed out earlier, but I believe that the benefits of an improved way to
>> manage upgrade routines will compensate by the breaking of compatibility.
>>
>> On Tue, Dec 29, 2015 at 10:25 AM, Wido den Hollander <wido@widodh.nl>
>> wrote:
>>
>> >
>> >
>> > On 29-12-15 13:21, Rafael Weingärtner wrote:
>> > > I got your point Daan.
>> > >
>> > > Well, and if we linked a version of ACS with a time stamp in the
>> format
>> > of
>> > > DD.MM.YYYY?
>> > >
>> >
>> > In that case you could also say.
>> >
>> > ACS 4.6.0 == db ver X
>> >
>> > You don't have to say ver >= X, you can also say ver = X.
>> >
>> > > We could then use the time stamp in the same format to name upgrade
>> > > routines. This way the idea of running all of the routines in between
>> > > version during upgrades could be applied.
>> > >
>> >
>> > Same goes for giving all database changes a simple numeric int which
>> > keeps incrementing each time a change is applied ;)
>> >
>> > Wido
>> >
>> > > On Tue, Dec 29, 2015 at 10:03 AM, Daan Hoogland <
>> daan.hoogland@gmail.com
>> > >
>> > > wrote:
>> > >
>> > >> Rafael,
>> > >>
>> > >> On Tue, Dec 29, 2015 at 12:22 PM, Rafael Weingärtner <
>> > >> rafaelweingartner@gmail.com> wrote:
>> > >>
>> > >>> Thanks, Daan and Wido for your contributions, I will discuss them
as
>> > >>> follows.
>> > >>>
>> > >>> Daan, about the idea of per commit upgrades. Do you mean that we
>> > separate
>> > >>> each change in the database that is introduced by PRs/Commits in
a
>> > >>> different file (routine upgrade) per ACS version?
>> > >>> So we would have, V_480_A.sql (for a PR),V_480_B.sql (for another
>> PR)
>> > and
>> > >>> so forth
>> > >>>
>> > >>> If that is the case, we can achieve that using a simple convention
>> > naming
>> > >>> as I suggested. Each developer when she/he needs to change or add
>> > >> something
>> > >>> in the database creates an upgrade routine separately and gives
it
>> an
>> > >>> execution order to be taken by Flywaydb. I think that could help
>> RMs to
>> > >>> track and isolate the problem, right?
>> > >>>
>> > >> ​Yes, with one little caveat. We do not know in what version a
>> > feature/PR
>> > >> will end up at the time of implementing, so a name containing the
>> > version
>> > >> would not be ideal.
>> > >> ​
>> > >>
>> > >>
>> > >>>
>> > >>> Hi Wido, now I understand your example.
>> > >>> I understand your worry about upgrade paths, and that is the point
I
>> > want
>> > >>> to discuss and solve. In your example, if we release a 4.6.0 and
>> later
>> > a
>> > >>> 4.5.3. You said that there would be no upgrade path from 4.5.3
to
>> > 4.6.0.
>> > >>> Well, today that is what happens. However, if we change the
>> technology
>> > we
>> > >>> use to upgrade the database (using a tool such as Flywaydb) and
if
>> we
>> > >>> define a standard to create upgrade routines that would not be
a
>> > problem.
>> > >>>
>> > >>> As I have written in my first email, to go from a version to
>> another we
>> > >>> should be able to run all of the upgrade routines in between them
>> > >>> (including the upgrade routine of the goal version). Therefore,
if
>> we
>> > >>> release a version 4.6.0, and then 4.5.3, if someone upgrades to
>> 4.5.3
>> > >> from
>> > >>> any other version, and then wants to upgrade to 4.6.0, that would
>> not
>> > be
>> > >> a
>> > >>> problem, it would be a metter of running only the routine upgrade
of
>> > >> 4.6.0
>> > >>> version. We do not need to explicitly create upgrade paths. They
>> should
>> > >> be
>> > >>> implicit by our upgrade conventions.
>> > >>>
>> > >>> About creating versions of the code that rely on some version of
the
>> > >>> database. I do not like much because of compatibility issues that
>> might
>> > >>> arise. For instance, let’s say version X of ACS depends on version
>> >=Y
>> > of
>> > >>> the database. If I upgrade the database to version Y + 1 or +2,
the
>> > same
>> > >>> ACS version has to keep running nice and shiny. My worry is that
may
>> > >> bring
>> > >>> some complications, such as to remove columns that cease to be
used
>> or
>> > >> data
>> > >>> structure that we might want to improve.
>> > >>>
>> > >>> I normally see that the database version and the code base are
tied
>> in
>> > a
>> > >>> mapping 1 to 1. Maybe I am having troubles identifying the benefits
>> of
>> > >> that
>> > >>> change.
>> > >>>
>> > >>> Thanks for your time ;)
>> > >>>
>> > >>> On Tue, Dec 29, 2015 at 8:15 AM, Wido den Hollander <wido@widodh.nl
>> >
>> > >>> wrote:
>> > >>>
>> > >>>>
>> > >>>>
>> > >>>> On 28-12-15 21:34, Rafael Weingärtner wrote:
>> > >>>>> Hi Wido, Rohit,
>> > >>>>> I have just read the feature suggestion.
>> > >>>>>
>> > >>>>> Wido, I am not trying to complicate things, quite the opposite,
I
>> > >> just
>> > >>>>> illustrate a simple thing that can happen and is happening;
I just
>> > >>>> pointed
>> > >>>>> how it can be easily solved.
>> > >>>>>
>> > >>>>> About the release of .Z, releases more constant and others,
I do
>> not
>> > >>> want
>> > >>>>> to mix topics. Let’s keep this thread strict to discuss
database
>> > >>>> upgrades.
>> > >>>>>
>> > >>>>
>> > >>>> I do not want to start the release discussion, but what I meant
is
>> > that
>> > >>>> we try to find a technical solution to something which might
be
>> solved
>> > >>>> easier by just changing the way we release.
>> > >>>>
>> > >>>> 4.6.0 is released and afterwards 4.5.3 is released. How does
>> somebody
>> > >>>> upgrade from 4.5.3 to 4.6.0? He can't, since the 4.6.0 code
doesn't
>> > >>>> support that path.
>> > >>>>
>> > >>>> So my idea is to split the database version from the code version.
>> > >>>>
>> > >>>> The code requires database version >= X and during boot
it simply
>> > >> checks
>> > >>>> that.
>> > >>>>
>> > >>>> The database migration tool can indeed do the DB migration,
it
>> doesn't
>> > >>>> have to be the mgmt server who does the upgrade.
>> > >>>>
>> > >>>>> Now, about the FS. I agree with Rohit that we should have
only one
>> > >> way
>> > >>> of
>> > >>>>> managing database upgrades and creation. I just do not
like the
>> idea
>> > >> of
>> > >>>>> creating a tool that work as a wrapper on frameworks/tools
such as
>> > >>>>> flywaydb. I think that those frameworks already work pretty
good
>> as
>> > >>> they
>> > >>>>> are; and, I would rather maintain configurations than some
wrapper
>> > >>> code.
>> > >>>>>
>> > >>>>> I personally like the way ACS works during upgrades (I
just do not
>> > >> like
>> > >>>> the
>> > >>>>> code itself and how things are structured), as a system
>> > >> administrator I
>> > >>>>> like to change the version in the
>> > >>>> “/etc/apt/sources.list.d/cloudstack.list”
>> > >>>>> and use the "apt-get" "update" and "install" from the command
>> line. I
>> > >>> do
>> > >>>>> not see the need to add another tool that is just a wrapper
to the
>> > >> mix.
>> > >>>> If
>> > >>>>> I update ACS code to 4.7.0, why would I let the database
schema
>> in an
>> > >>>> older
>> > >>>>> version? And if we want version DB schemas and application
code
>> > >>>> separately
>> > >>>>> maintaining somehow compatibility between them, which would
bring
>> a
>> > >>> whole
>> > >>>>> other level of complexity to the code; I think we should
avoid
>> that.
>> > >>>>>
>> > >>>>> The flywaydb can be easily integrated with everything we
have
>> now; we
>> > >>>> could
>> > >>>>> have a maven profile for developers and integrate it in
ACS
>> bootstrap
>> > >>>> using
>> > >>>>> its API as a Spring bean. Therefore, we could remove the
current
>> > >>>>> “DatabaseUpgradeChecker “, “DbUpgrade” and other
classes that aim
>> to
>> > >> do
>> > >>>>> that. We could even add the creation of the schema into
the first
>> > >> time
>> > >>> it
>> > >>>>> boots using flywaydb and retire the “cloudstack-setup-database”
>> > >> script,
>> > >>>> or
>> > >>>>> at least make it less complicated, using it just to configure
the
>> > >>>> database
>> > >>>>> URL and users.
>> > >>>>>
>> > >>>>> The point is that to use Flywaydb we would have to agree
upon a
>> > >>>> convention
>> > >>>>> on creating routines (java and SQL) to execute upgrades.
Moreover,
>> > >>> using
>> > >>>> a
>> > >>>>> tool such as Flywaydb we do not need to worry about upgrade
paths.
>> > >> As I
>> > >>>>> wrote in the email I used to start this thread, the upgrade
has
>> to be
>> > >>>>> straightforward, to go to a version we have to run all
of the
>> upgrade
>> > >>>>> routines between the current version until the desired
one. Our
>> job
>> > >> is
>> > >>> to
>> > >>>>> create upgrade routines that work and name them properly,
the job
>> of
>> > >>> the
>> > >>>>> tool is to check the current version, the desired one,
the
>> upgrades
>> > >>> that
>> > >>>> it
>> > >>>>> needs to run and execute everything properly.
>> > >>>>>
>> > >>>>
>> > >>>> Yes, indeed. I just wanted to start the discussion if we shouldn't
>> > >>>> version the database differently from the code.
>> > >>>>
>> > >>>>> Additionally, I do not see the need to break compatibility
as
>> Rohit
>> > >>>>> suggested in the FS; in my opinion, everything we have
up today
>> can
>> > >> be
>> > >>>>> migrated to the new structure I proposed. If we use a tool
such as
>> > >>>>> Flywaydb, I even volunteered for that. The only thing we
have to
>> > >>> discuss
>> > >>>>> and agree upon is the naming conventions for upgrades routines,
>> where
>> > >>> to
>> > >>>>> put them and the configurations for flywaydb.
>> > >>>>>
>> > >>>>> Thanks for your contribution and time.
>> > >>>>>
>> > >>>>>
>> > >>>>> On Mon, Dec 28, 2015 at 2:10 PM, Rohit Yadav <
>> > >>> rohit.yadav@shapeblue.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi Rafael and Wido,
>> > >>>>>>
>> > >>>>>> Thanks for starting a conversation in this regard,
I could not
>> > >> pursue
>> > >>>> the
>> > >>>>>> Chimp tool due to other $dayjob work though it’s
good to see some
>> > >>>>>> discussion has started again. Hope we’ll solve this
in 2016.
>> > >>>>>>
>> > >>>>>> In my opinion, we will need to first separate the database
>> > >>>> init/migration
>> > >>>>>> tooling away from mgmt server (right now the mgmt server
does db
>> > >>>> migrations
>> > >>>>>> when it starts and there is a code/db version mismatch)
and
>> secondly
>> > >>>> make
>> > >>>>>> sure that we’re using the same code/tool to deploy
database
>> (right
>> > >>> now,
>> > >>>>>> users use the cloudstack-setup-database python tool
while
>> developer
>> > >>> use
>> > >>>> the
>> > >>>>>> maven/java DatabaseCreator activated by the -Ddeploydb
flag).
>> > >>>>>>
>> > >>>>>> After we’ve addressed these two issues we can look
into how we
>> can
>> > >>>> support
>> > >>>>>> minor releases workflow (or decide to do something
else, like not
>> > >>>> support
>> > >>>>>> .Z releases like Wido mentioned), and see if we can
or want to
>> use
>> > >> any
>> > >>>>>> existing migration tool or write a wrapper tool “chimp”
that uses
>> > >>>> existing
>> > >>>>>> tools (some of those are mentioned in the Chimp FS
like flywaydb
>> > >> etc).
>> > >>>> For
>> > >>>>>> allowing users to go back and forth from a db schema/version,
>> we’ll
>> > >>> also
>> > >>>>>> need some new DB migration
>> > >>> conventions/versioning/rules/static-checking,
>> > >>>>>> and how developer need to write such paths (forward
and reverse)
>> > >> etc.
>> > >>>>>>
>> > >>>>>> The best approach I figured at the time was to decide
that we’ll
>> use
>> > >>> the
>> > >>>>>> previous db upgrade path mechanism till a certain CloudStack
>> version
>> > >>>> (say
>> > >>>>>> 4.8.0) and after that we’ll use the new approach
or tooling to
>> > >>>>>> upgrade/downgrade DB schemas (thereby retiring away
from the old
>> DB
>> > >>>> upgrade
>> > >>>>>> path mess).
>> > >>>>>>
>> > >>>>>>>
>> > >>>>>>
>> > >>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
Rohit Yadav
>> Software
>> > >>>>>> Architect ,  ShapeBlue d:  * | s: +44 203 603 0540*
>> > >>>>>> <%7C%20s:%20+44%20203%20603%200540>  |  m:  *+91
8826230892*
>> > >>>>>> <+91%208826230892> e:  *rohit.yadav@shapeblue.com
| t: *
>> > >>>>>> <rohit.yadav@shapeblue.com%20%7C%20t:>  |  w:
 *
>> www.shapeblue.com*
>> > >>>>>> <http://www.shapeblue.com> a:
>> > >>>>>> 53 Chandos Place, Covent Garden London WC2N 4HS UK
Shape Blue Ltd
>> > >> is a
>> > >>>>>> company incorporated in England & Wales. ShapeBlue
Services India
>> > >> LLP
>> > >>>> is a
>> > >>>>>> company incorporated in India and is operated under
license from
>> > >> Shape
>> > >>>> Blue
>> > >>>>>> Ltd. Shape Blue Brasil Consultoria Ltda is a company
>> incorporated in
>> > >>>> Brasil
>> > >>>>>> and is operated under license from Shape Blue Ltd.
ShapeBlue SA
>> Pty
>> > >>> Ltd
>> > >>>> is
>> > >>>>>> a company registered by The Republic of South Africa
and is
>> traded
>> > >>> under
>> > >>>>>> license from Shape Blue Ltd. ShapeBlue is a registered
trademark.
>> > >>>>>> This email and any attachments to it may be confidential
and are
>> > >>>> intended
>> > >>>>>> solely for the use of the individual to whom it is
addressed. Any
>> > >>> views
>> > >>>> or
>> > >>>>>> opinions expressed are solely those of the author and
do not
>> > >>> necessarily
>> > >>>>>> represent those of Shape Blue Ltd or related companies.
If you
>> are
>> > >> not
>> > >>>> the
>> > >>>>>> intended recipient of this email, you must neither
take any
>> action
>> > >>> based
>> > >>>>>> upon its contents, nor copy or show it to anyone. Please
contact
>> the
>> > >>>> sender
>> > >>>>>> if you believe you have received this email in error.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On 28-Dec-2015, at 9:10 PM, Wido den Hollander <wido@widodh.nl>
>> > >>> wrote:
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> On 28-12-15 16:21, Rafael Weingärtner wrote:
>> > >>>>>>>> Thanks for your contribution Wido,
>> > >>>>>>>> I have not seen Rohit’s email; I will take
a look at it.
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> Ok, he has a FS here:
>> > >>>>>>>
>> > >>>>
>> > >>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Chimp
>> > >>>>>>>
>> > >>>>>>>> About database schema changes happening only
in X.Y, I also
>> agree
>> > >>> with
>> > >>>>>> you
>> > >>>>>>>> (that is a convention we all could agree on,
and such as
>> conding
>> > >> and
>> > >>>>>>>> release procedures we could have a wiki page
for that).
>> However, I
>> > >>>>>> think we
>> > >>>>>>>> still might have scripts in versions X.Y.Z
to add data to a
>> table
>> > >>> such
>> > >>>>>> as
>> > >>>>>>>> “guest_os_hypervisor”.
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> Yes, that is true. A bugfix could be a addition
into the
>> database,
>> > >>> but
>> > >>>>>>> we have to prevent it as much as possible.
>> > >>>>>>>
>> > >>>>>>>> The point to manage such scripts is that, if
we are in version
>> > >> such
>> > >>> as
>> > >>>>>>>> 4.7.0 and a new script emerges in version 4.5.3,
we would have
>> to
>> > >>>>>> decide to
>> > >>>>>>>> run or not to run it. I would rather not run
them, since if
>> they
>> > >> add
>> > >>>>>>>> something to the code base; those changes should
also be
>> applied
>> > >>> into
>> > >>>>>>>> master and as a consequence it will be available
in a future
>> > >> update.
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> I understand, but this is where our release cycle
becomes the
>> > >>> problem.
>> > >>>>>>> It is because we release a X.Y.Z release we run
into these kind
>> of
>> > >>>>>> problems.
>> > >>>>>>>
>> > >>>>>>> If we as a project simple do not release the .Z
releases we
>> would
>> > >> be
>> > >>>>>>> fine as well ;)
>> > >>>>>>>
>> > >>>>>>> You can try to complicate things with technical
things, or if we
>> > >>>> release
>> > >>>>>>> every two / three weeks we don't run into these
kind of
>> situations
>> > >> :)
>> > >>>>>>>
>> > >>>>>>> We might even cut the database version loose from
the code
>> version.
>> > >>>>>>>
>> > >>>>>>> Database version is simple 100, 101, 102, 103,
104, 105. And a
>> code
>> > >>>>>>> version requires a certain version of the database.
>> > >>>>>>>
>> > >>>>>>> Wido
>> > >>>>>>>
>> > >>>>>>>> On Mon, Dec 28, 2015 at 12:50 PM, Wido den
Hollander <
>> > >>> wido@widodh.nl>
>> > >>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On 28-12-15 14:16, Rafael Weingärtner
wrote:
>> > >>>>>>>>>> Hi all devs,
>> > >>>>>>>>>> First of all, sorry the long text,
but I hope we can start a
>> > >>>>>> discussion
>> > >>>>>>>>>> here and improve that part of ACS.
>> > >>>>>>>>>>
>> > >>>>>>>>>> A while ago I have faced the code that
Apache CloudStack
>> (ACS)
>> > >>> uses
>> > >>>> to
>> > >>>>>>>>>> upgrade from a version to newer one
and that did not seem to
>> be
>> > >> a
>> > >>>> good
>> > >>>>>>>>> way
>> > >>>>>>>>>> to execute our upgrades. Therefore,
I decided to use some
>> time
>> > >> to
>> > >>>>>> search
>> > >>>>>>>>>> for alternatives.
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> I think we all saw that happen once or
more :)
>> > >>>>>>>>>
>> > >>>>>>>>>> I have read some material about versioning
of scripts used to
>> > >>>> upgrade
>> > >>>>>> a
>> > >>>>>>>>>> database (DB) of a system and went
through some frameworks
>> that
>> > >>>> could
>> > >>>>>>>>> help
>> > >>>>>>>>>> us.
>> > >>>>>>>>>>
>> > >>>>>>>>>> In the literature of software engineering,
it is firmly
>> stated
>> > >>> that
>> > >>>> we
>> > >>>>>>>>> have
>> > >>>>>>>>>> to version DB scripts as we do with
the source code of the
>> > >>>>>> application,
>> > >>>>>>>>>> using the baseline approach. Gladly,
we were not that bad at
>> > >> this
>> > >>>>>> point,
>> > >>>>>>>>> we
>> > >>>>>>>>>> already versioned our routines for
DB upgrade (.sql and
>> .java).
>> > >>>>>>>>> Therefore,
>> > >>>>>>>>>> it seemed that we just did not have
used a practical
>> approach to
>> > >>>> help
>> > >>>>>> us
>> > >>>>>>>>>> during DB upgrades.
>> > >>>>>>>>>>
>> > >>>>>>>>>> From my readings and looking at the
ACS source code I raised
>> the
>> > >>>>>>>>> following
>> > >>>>>>>>>> requirement:
>> > >>>>>>>>>> • We should be able to write more
than one routine to upgrade
>> > >> to a
>> > >>>>>>>>>> version; those routines can be written
in Java and SQL. We
>> might
>> > >>>> have
>> > >>>>>>>>> more
>> > >>>>>>>>>> than a routine to be executed for each
version and we should
>> be
>> > >>> able
>> > >>>>>> to
>> > >>>>>>>>>> define an order of execution. Additionally,
to go to an upper
>> > >>>>>> version, we
>> > >>>>>>>>>> have to run all of the routines from
smaller versions first,
>> > >> until
>> > >>>> we
>> > >>>>>>>>>> achieve the desired version.
>> > >>>>>>>>>>
>> > >>>>>>>>>> We could also add another requirement
that is the downgrade
>> > >> from a
>> > >>>>>>>>> version,
>> > >>>>>>>>>> which we currently do not support.
With that comes my first
>> > >>> question
>> > >>>>>> for
>> > >>>>>>>>>> discussion:
>> > >>>>>>>>>> • Do we want/need a method to downgrade
from a version to a
>> > >>> previous
>> > >>>>>>>>> one?
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> I personally do not care. Usually people
should create a
>> backup
>> > >>> PRIOR
>> > >>>>>> to
>> > >>>>>>>>> a upgrade. If that fails they can restore
the backup.
>> > >>>>>>>>>
>> > >>>>>>>>>> I found an explanation for not supporting
downgrades, and I
>> > >> liked
>> > >>>> it:
>> > >>>>>>>>>> http://flywaydb.org/documentation/faq.html#downgrade
>> > >>>>>>>>>>
>> > >>>>>>>>>> So, what I devised for us:
>> > >>>>>>>>>> First the bureaucracy part - our migrations
occur basically
>> in
>> > >>> three
>> > >>>>>> (3)
>> > >>>>>>>>>> steps, first we have a "prepare script",
then a cleanup
>> script
>> > >> and
>> > >>>>>>>>> finally
>> > >>>>>>>>>> the migration per se that is written
in Java, at least, that
>> is
>> > >>> what
>> > >>>>>> we
>> > >>>>>>>>> can
>> > >>>>>>>>>> expect when reading the interface
>> > >>> “com.cloud.upgrade.dao.DbUpgrade”.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Additionally, our scripts have the
following naming
>> convention:
>> > >>>>>>>>>> schema-<currentVersion>to<desiredVersion>,
which in IMHO may
>> > >> cause
>> > >>>>>> some
>> > >>>>>>>>>> confusion because at first sight we
may think that from the
>> same
>> > >>>>>> version
>> > >>>>>>>>> we
>> > >>>>>>>>>> could have different paths to an upper
version, which in
>> > >> practice
>> > >>> is
>> > >>>>>> not
>> > >>>>>>>>>> happening. Instead of a <currentVersion>to<version>
we could
>> > >>> simply
>> > >>>>>> use
>> > >>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension>,
giving
>> that,
>> > >> we
>> > >>>>>> have to
>> > >>>>>>>>>> execute all of the V_<version>
scripts that are smaller than
>> the
>> > >>>>>> version
>> > >>>>>>>>> we
>> > >>>>>>>>>> want to upgrade.
>> > >>>>>>>>>>
>> > >>>>>>>>>> To clarify what I am saying, I will
use an example. Let’s
>> say we
>> > >>>> have
>> > >>>>>>>>> just
>> > >>>>>>>>>> installed ACS and ran the cloudstack-setup-database.
That
>> > >> command
>> > >>>> will
>> > >>>>>>>>>> create a database schema in version
4.0.0. To upgrade that
>> > >> schema
>> > >>> to
>> > >>>>>>>>>> version 4.3.0 (it is just an example,
it could be any other
>> > >>>> version),
>> > >>>>>> ACS
>> > >>>>>>>>>> will use the following mapping:
>> > >>>>>>>>>>
>> > >>>>>>>>>> _upgradeMap.put("4.0.0", new DbUpgrade[]
{new
>> Upgrade40to41(),
>> > >> new
>> > >>>>>>>>>> Upgrade410to420(), new Upgrade420to421(),
new
>> Upgrade421to430())
>> > >>>>>>>>>>
>> > >>>>>>>>>> After loading the mapping, ACS will
execute the scripts
>> defined
>> > >> in
>> > >>>>>> each
>> > >>>>>>>>> one
>> > >>>>>>>>>> of the Upgrade path classes and the
migration code per se.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Now, let’s say we change the “.sql”
scripts name to the
>> pattern
>> > >> I
>> > >>>>>>>>>> mentioned, we would have the following
scripts; those are the
>> > >>>> scripts
>> > >>>>>>>>> found
>> > >>>>>>>>>> that aim to upgrade to versions between
the interval 4.0.0 –
>> > >> 4.3.0
>> > >>>>>>>>>> (considering 4.3.0, since that is the
goal version):
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> - schema-40to410, can be named to:
V_410_A.sql
>> > >>>>>>>>>> - schema-40to410-cleanup, can be named
to: V_410_B.sql
>> > >>>>>>>>>> - schema-410to420, can be named to:
V_420_A.sql
>> > >>>>>>>>>> - schema-410to420-cleanup , can be
named to: V_420_b.sql
>> > >>>>>>>>>> - schema-420to421, can be named to:
V_421_A.sql
>> > >>>>>>>>>> - schema-421to430, can be named to:
V_430_A.sql
>> > >>>>>>>>>> - schema-421to430-cleanup, can be named
to: V_430_B.sql
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> Additionally, all of the java code
would have to follow the
>> same
>> > >>>>>>>>>> convention. For instance, we have
>> > >>>>>> “com.cloud.upgrade.dao.Upgrade40to41”,
>> > >>>>>>>>>> which has some java code to migrate
from 4.0.0 to 4.1.0. The
>> > >> idea
>> > >>> is
>> > >>>>>> to
>> > >>>>>>>>>> extract that migration code to a Java
class named:
>> V_410_C.java,
>> > >>>>>> giving
>> > >>>>>>>>>> that it has to execute the SQL scripts
before the java code.
>> > >>>>>>>>>>
>> > >>>>>>>>>> In order to go from a smaller version
(4.0.0) to an upper one
>> > >>>>>> (4.3.0), we
>> > >>>>>>>>>> have to run all of the migration routines
from intermediate
>> > >>>> versions.
>> > >>>>>>>>> That
>> > >>>>>>>>>> is what we are already doing, but we
do all of that manually.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Bottom line, I think we could simple
use the convention
>> > >>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension>
to name
>> upgrade
>> > >>>>>>>>> routines.
>> > >>>>>>>>>> That would facilitate us to use a framework
to help us with
>> that
>> > >>>>>> process.
>> > >>>>>>>>>> Additionally, I believe that we should
always assume that to
>> go
>> > >>>> from a
>> > >>>>>>>>>> smaller version to a higher one, we
should run all of the
>> > >> scripts
>> > >>>> that
>> > >>>>>>>>>> exist between them. What do you guys
think of that?
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> That seems good to me. But we still have
to prevent that we
>> > >> perform
>> > >>>>>>>>> database changes in a X.Y.Z release since
that is branched off
>> > >> to a
>> > >>>>>>>>> different branch.
>> > >>>>>>>>>
>> > >>>>>>>>> Imho database changes should only happen
in X.Y releases.
>> > >>>>>>>>>
>> > >>>>>>>>>> After the bureaucracy, we can discuss
tools. If we use that
>> > >>>>>> convention to
>> > >>>>>>>>>> name migration (upgrade) routines,
we can start thinking on
>> > >> tools
>> > >>> to
>> > >>>>>>>>>> support our migration process. I found
two (2) promising
>> ones:
>> > >>>>>> Liquibase
>> > >>>>>>>>>> and Flywaydb (both seem to be under
Apache license, but the
>> > >> first
>> > >>>> one
>> > >>>>>> has
>> > >>>>>>>>>> an enterprise version?!). After reading
the documentation and
>> > >> some
>> > >>>>>> usage
>> > >>>>>>>>>> examples I found the flywaydb easier
and simpler to use.
>> > >>>>>>>>>>
>> > >>>>>>>>>> What are the options of tools that
we can use to help us
>> manage
>> > >>> the
>> > >>>>>>>>>> database upgrade, without needing to
code the upgrade path
>> that
>> > >>> you
>> > >>>>>> know?
>> > >>>>>>>>>>
>> > >>>>>>>>>> After that, I think we should decide
if we should create
>> another
>> > >>>>>>>>>> project/component to take care of migrations,
or we can just
>> add
>> > >>> the
>> > >>>>>>>>>> dependency of the tool to a project
such as
>> “cloud-framework-db”
>> > >>> and
>> > >>>>>>>>> start
>> > >>>>>>>>>> using it.
>> > >>>>>>>>>>
>> > >>>>>>>>>> The “cloud-framework-db” project
seems to have a focus on
>> other
>> > >>>> things
>> > >>>>>>>>> such
>> > >>>>>>>>>> as managing transactions and generating
SQLs from annotations
>> > >> (?!?
>> > >>>>>> That
>> > >>>>>>>>>> should be a topic for another discussion).
Therefore, I would
>> > >>> rather
>> > >>>>>>>>> create
>> > >>>>>>>>>> a new project that has the specific
goal of managing ACS DB
>> > >>>> upgrades.
>> > >>>>>> I
>> > >>>>>>>>>> would also move all of the routines
(SQL and Java) to this
>> new
>> > >>>>>> project.
>> > >>>>>>>>>> This project would be a module of the
CloudStack project and
>> it
>> > >>>> would
>> > >>>>>>>>>> execute the upgrade routines at the
startup of ACS.
>> > >>>>>>>>>>
>> > >>>>>>>>>> I believe that going from a homemade
solution to one that is
>> > >> more
>> > >>>>>>>>>> consolidated and used by other communities
would be the way
>> to
>> > >> go.
>> > >>>>>>>>>>
>> > >>>>>>>>>> I can volunteer myself to create a
PR with the aforementioned
>> > >>>> changes
>> > >>>>>> and
>> > >>>>>>>>>> using flywaydb to manage our upgrades.
However, I prefer to
>> > >> have a
>> > >>>>>> good
>> > >>>>>>>>>> discussion with other devs first, before
starting coding.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Do you have suggestions or points that
should be raised
>> before
>> > >> we
>> > >>>>>> start
>> > >>>>>>>>>> working on that?
>> > >>>>>>>>>
>> > >>>>>>>>> Rohit suggested Chimp earlier this year:
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>
>> >
>> http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201508.mbox/%3C677BD09F-FC75-4888-8DC8-2B7AF7439221@shapeblue.com%3E
>> > >>>>>>>>>
>> > >>>>>>>>> The thread is called: "[DISCUSS] Let's
fix CloudStack Upgrades
>> > >> and
>> > >>> DB
>> > >>>>>>>>> migrations with CloudStack Chimp"
>> > >>>>>>>>>
>> > >>>>>>>>> Maybe there is something good in there.
>> > >>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> --
>> > >>>>>>>>>> Rafael Weingärtner
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>>> Regards.
>> > >>>>>>
>> > >>>>>> Find out more about ShapeBlue and our range of CloudStack
related
>> > >>>> services:
>> > >>>>>> IaaS Cloud Design & Build
>> > >>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//>
| CSForge –
>> > >>> rapid
>> > >>>>>> IaaS deployment framework <http://shapeblue.com/csforge/>
>> > >>>>>> CloudStack Consulting <
>> http://shapeblue.com/cloudstack-consultancy/
>> > >>>
>> > >>> |
>> > >>>> CloudStack
>> > >>>>>> Software Engineering
>> > >>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>> > >>>>>> CloudStack Infrastructure Support
>> > >>>>>> <http://shapeblue.com/cloudstack-infrastructure-support/>
|
>> > >>> CloudStack
>> > >>>>>> Bootcamp Training Courses <
>> > >> http://shapeblue.com/cloudstack-training/>
>> > >>>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> --
>> > >>> Rafael Weingärtner
>> > >>>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Daan
>> > >>
>> > >
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Rafael Weingärtner
>>
>
>
>
> --
> Daan
>



-- 
Daan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message