cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Weingärtner <rafaelweingart...@gmail.com>
Subject Re: Let’s discuss database upgrades
Date Tue, 29 Dec 2015 12:40:46 GMT
Wido, that is true, you are right; the naming on upgrade routines can use a
numeric value independent of the number of the version. The numeric value
can be a simple integer that is incremented each routine that is added or a
time stamp when the routine was added. The point is that we would have to
link a version to a number. That would enable us to use flywaydb.

To use that approach I think we might need to break compatibility as you
pointed out earlier, but I believe that the benefits of an improved way to
manage upgrade routines will compensate by the breaking of compatibility.

On Tue, Dec 29, 2015 at 10:25 AM, Wido den Hollander <wido@widodh.nl> wrote:

>
>
> On 29-12-15 13:21, Rafael Weingärtner wrote:
> > I got your point Daan.
> >
> > Well, and if we linked a version of ACS with a time stamp in the format
> of
> > DD.MM.YYYY?
> >
>
> In that case you could also say.
>
> ACS 4.6.0 == db ver X
>
> You don't have to say ver >= X, you can also say ver = X.
>
> > We could then use the time stamp in the same format to name upgrade
> > routines. This way the idea of running all of the routines in between
> > version during upgrades could be applied.
> >
>
> Same goes for giving all database changes a simple numeric int which
> keeps incrementing each time a change is applied ;)
>
> Wido
>
> > On Tue, Dec 29, 2015 at 10:03 AM, Daan Hoogland <daan.hoogland@gmail.com
> >
> > wrote:
> >
> >> Rafael,
> >>
> >> On Tue, Dec 29, 2015 at 12:22 PM, Rafael Weingärtner <
> >> rafaelweingartner@gmail.com> wrote:
> >>
> >>> Thanks, Daan and Wido for your contributions, I will discuss them as
> >>> follows.
> >>>
> >>> Daan, about the idea of per commit upgrades. Do you mean that we
> separate
> >>> each change in the database that is introduced by PRs/Commits in a
> >>> different file (routine upgrade) per ACS version?
> >>> So we would have, V_480_A.sql (for a PR),V_480_B.sql (for another PR)
> and
> >>> so forth
> >>>
> >>> If that is the case, we can achieve that using a simple convention
> naming
> >>> as I suggested. Each developer when she/he needs to change or add
> >> something
> >>> in the database creates an upgrade routine separately and gives it an
> >>> execution order to be taken by Flywaydb. I think that could help RMs to
> >>> track and isolate the problem, right?
> >>>
> >> ​Yes, with one little caveat. We do not know in what version a
> feature/PR
> >> will end up at the time of implementing, so a name containing the
> version
> >> would not be ideal.
> >> ​
> >>
> >>
> >>>
> >>> Hi Wido, now I understand your example.
> >>> I understand your worry about upgrade paths, and that is the point I
> want
> >>> to discuss and solve. In your example, if we release a 4.6.0 and later
> a
> >>> 4.5.3. You said that there would be no upgrade path from 4.5.3 to
> 4.6.0.
> >>> Well, today that is what happens. However, if we change the technology
> we
> >>> use to upgrade the database (using a tool such as Flywaydb) and if we
> >>> define a standard to create upgrade routines that would not be a
> problem.
> >>>
> >>> As I have written in my first email, to go from a version to another we
> >>> should be able to run all of the upgrade routines in between them
> >>> (including the upgrade routine of the goal version). Therefore, if we
> >>> release a version 4.6.0, and then 4.5.3, if someone upgrades to 4.5.3
> >> from
> >>> any other version, and then wants to upgrade to 4.6.0, that would not
> be
> >> a
> >>> problem, it would be a metter of running only the routine upgrade of
> >> 4.6.0
> >>> version. We do not need to explicitly create upgrade paths. They should
> >> be
> >>> implicit by our upgrade conventions.
> >>>
> >>> About creating versions of the code that rely on some version of the
> >>> database. I do not like much because of compatibility issues that might
> >>> arise. For instance, let’s say version X of ACS depends on version >=Y
> of
> >>> the database. If I upgrade the database to version Y + 1 or +2, the
> same
> >>> ACS version has to keep running nice and shiny. My worry is that may
> >> bring
> >>> some complications, such as to remove columns that cease to be used or
> >> data
> >>> structure that we might want to improve.
> >>>
> >>> I normally see that the database version and the code base are tied in
> a
> >>> mapping 1 to 1. Maybe I am having troubles identifying the benefits of
> >> that
> >>> change.
> >>>
> >>> Thanks for your time ;)
> >>>
> >>> On Tue, Dec 29, 2015 at 8:15 AM, Wido den Hollander <wido@widodh.nl>
> >>> wrote:
> >>>
> >>>>
> >>>>
> >>>> On 28-12-15 21:34, Rafael Weingärtner wrote:
> >>>>> Hi Wido, Rohit,
> >>>>> I have just read the feature suggestion.
> >>>>>
> >>>>> Wido, I am not trying to complicate things, quite the opposite,
I
> >> just
> >>>>> illustrate a simple thing that can happen and is happening; I just
> >>>> pointed
> >>>>> how it can be easily solved.
> >>>>>
> >>>>> About the release of .Z, releases more constant and others, I do
not
> >>> want
> >>>>> to mix topics. Let’s keep this thread strict to discuss database
> >>>> upgrades.
> >>>>>
> >>>>
> >>>> I do not want to start the release discussion, but what I meant is
> that
> >>>> we try to find a technical solution to something which might be solved
> >>>> easier by just changing the way we release.
> >>>>
> >>>> 4.6.0 is released and afterwards 4.5.3 is released. How does somebody
> >>>> upgrade from 4.5.3 to 4.6.0? He can't, since the 4.6.0 code doesn't
> >>>> support that path.
> >>>>
> >>>> So my idea is to split the database version from the code version.
> >>>>
> >>>> The code requires database version >= X and during boot it simply
> >> checks
> >>>> that.
> >>>>
> >>>> The database migration tool can indeed do the DB migration, it doesn't
> >>>> have to be the mgmt server who does the upgrade.
> >>>>
> >>>>> Now, about the FS. I agree with Rohit that we should have only one
> >> way
> >>> of
> >>>>> managing database upgrades and creation. I just do not like the
idea
> >> of
> >>>>> creating a tool that work as a wrapper on frameworks/tools such
as
> >>>>> flywaydb. I think that those frameworks already work pretty good
as
> >>> they
> >>>>> are; and, I would rather maintain configurations than some wrapper
> >>> code.
> >>>>>
> >>>>> I personally like the way ACS works during upgrades (I just do not
> >> like
> >>>> the
> >>>>> code itself and how things are structured), as a system
> >> administrator I
> >>>>> like to change the version in the
> >>>> “/etc/apt/sources.list.d/cloudstack.list”
> >>>>> and use the "apt-get" "update" and "install" from the command line.
I
> >>> do
> >>>>> not see the need to add another tool that is just a wrapper to the
> >> mix.
> >>>> If
> >>>>> I update ACS code to 4.7.0, why would I let the database schema
in an
> >>>> older
> >>>>> version? And if we want version DB schemas and application code
> >>>> separately
> >>>>> maintaining somehow compatibility between them, which would bring
a
> >>> whole
> >>>>> other level of complexity to the code; I think we should avoid that.
> >>>>>
> >>>>> The flywaydb can be easily integrated with everything we have now;
we
> >>>> could
> >>>>> have a maven profile for developers and integrate it in ACS bootstrap
> >>>> using
> >>>>> its API as a Spring bean. Therefore, we could remove the current
> >>>>> “DatabaseUpgradeChecker “, “DbUpgrade” and other classes
that aim to
> >> do
> >>>>> that. We could even add the creation of the schema into the first
> >> time
> >>> it
> >>>>> boots using flywaydb and retire the “cloudstack-setup-database”
> >> script,
> >>>> or
> >>>>> at least make it less complicated, using it just to configure the
> >>>> database
> >>>>> URL and users.
> >>>>>
> >>>>> The point is that to use Flywaydb we would have to agree upon a
> >>>> convention
> >>>>> on creating routines (java and SQL) to execute upgrades. Moreover,
> >>> using
> >>>> a
> >>>>> tool such as Flywaydb we do not need to worry about upgrade paths.
> >> As I
> >>>>> wrote in the email I used to start this thread, the upgrade has
to be
> >>>>> straightforward, to go to a version we have to run all of the upgrade
> >>>>> routines between the current version until the desired one. Our
job
> >> is
> >>> to
> >>>>> create upgrade routines that work and name them properly, the job
of
> >>> the
> >>>>> tool is to check the current version, the desired one, the upgrades
> >>> that
> >>>> it
> >>>>> needs to run and execute everything properly.
> >>>>>
> >>>>
> >>>> Yes, indeed. I just wanted to start the discussion if we shouldn't
> >>>> version the database differently from the code.
> >>>>
> >>>>> Additionally, I do not see the need to break compatibility as Rohit
> >>>>> suggested in the FS; in my opinion, everything we have up today
can
> >> be
> >>>>> migrated to the new structure I proposed. If we use a tool such
as
> >>>>> Flywaydb, I even volunteered for that. The only thing we have to
> >>> discuss
> >>>>> and agree upon is the naming conventions for upgrades routines,
where
> >>> to
> >>>>> put them and the configurations for flywaydb.
> >>>>>
> >>>>> Thanks for your contribution and time.
> >>>>>
> >>>>>
> >>>>> On Mon, Dec 28, 2015 at 2:10 PM, Rohit Yadav <
> >>> rohit.yadav@shapeblue.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi Rafael and Wido,
> >>>>>>
> >>>>>> Thanks for starting a conversation in this regard, I could not
> >> pursue
> >>>> the
> >>>>>> Chimp tool due to other $dayjob work though it’s good to see
some
> >>>>>> discussion has started again. Hope we’ll solve this in 2016.
> >>>>>>
> >>>>>> In my opinion, we will need to first separate the database
> >>>> init/migration
> >>>>>> tooling away from mgmt server (right now the mgmt server does
db
> >>>> migrations
> >>>>>> when it starts and there is a code/db version mismatch) and
secondly
> >>>> make
> >>>>>> sure that we’re using the same code/tool to deploy database
(right
> >>> now,
> >>>>>> users use the cloudstack-setup-database python tool while developer
> >>> use
> >>>> the
> >>>>>> maven/java DatabaseCreator activated by the -Ddeploydb flag).
> >>>>>>
> >>>>>> After we’ve addressed these two issues we can look into how
we can
> >>>> support
> >>>>>> minor releases workflow (or decide to do something else, like
not
> >>>> support
> >>>>>> .Z releases like Wido mentioned), and see if we can or want
to use
> >> any
> >>>>>> existing migration tool or write a wrapper tool “chimp”
that uses
> >>>> existing
> >>>>>> tools (some of those are mentioned in the Chimp FS like flywaydb
> >> etc).
> >>>> For
> >>>>>> allowing users to go back and forth from a db schema/version,
we’ll
> >>> also
> >>>>>> need some new DB migration
> >>> conventions/versioning/rules/static-checking,
> >>>>>> and how developer need to write such paths (forward and reverse)
> >> etc.
> >>>>>>
> >>>>>> The best approach I figured at the time was to decide that we’ll
use
> >>> the
> >>>>>> previous db upgrade path mechanism till a certain CloudStack
version
> >>>> (say
> >>>>>> 4.8.0) and after that we’ll use the new approach or tooling
to
> >>>>>> upgrade/downgrade DB schemas (thereby retiring away from the
old DB
> >>>> upgrade
> >>>>>> path mess).
> >>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> [image: ShapeBlue] <http://www.shapeblue.com> Rohit Yadav
Software
> >>>>>> Architect ,  ShapeBlue d:  * | s: +44 203 603 0540*
> >>>>>> <%7C%20s:%20+44%20203%20603%200540>  |  m:  *+91 8826230892*
> >>>>>> <+91%208826230892> e:  *rohit.yadav@shapeblue.com | t:
*
> >>>>>> <rohit.yadav@shapeblue.com%20%7C%20t:>  |  w:  *www.shapeblue.com*
> >>>>>> <http://www.shapeblue.com> a:
> >>>>>> 53 Chandos Place, Covent Garden London WC2N 4HS UK Shape Blue
Ltd
> >> is a
> >>>>>> company incorporated in England & Wales. ShapeBlue Services
India
> >> LLP
> >>>> is a
> >>>>>> company incorporated in India and is operated under license
from
> >> Shape
> >>>> Blue
> >>>>>> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated
in
> >>>> Brasil
> >>>>>> and is operated under license from Shape Blue Ltd. ShapeBlue
SA Pty
> >>> Ltd
> >>>> is
> >>>>>> a company registered by The Republic of South Africa and is
traded
> >>> under
> >>>>>> license from Shape Blue Ltd. ShapeBlue is a registered trademark.
> >>>>>> This email and any attachments to it may be confidential and
are
> >>>> intended
> >>>>>> solely for the use of the individual to whom it is addressed.
Any
> >>> views
> >>>> or
> >>>>>> opinions expressed are solely those of the author and do not
> >>> necessarily
> >>>>>> represent those of Shape Blue Ltd or related companies. If you
are
> >> not
> >>>> the
> >>>>>> intended recipient of this email, you must neither take any
action
> >>> based
> >>>>>> upon its contents, nor copy or show it to anyone. Please contact
the
> >>>> sender
> >>>>>> if you believe you have received this email in error.
> >>>>>>
> >>>>>>
> >>>>>> On 28-Dec-2015, at 9:10 PM, Wido den Hollander <wido@widodh.nl>
> >>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 28-12-15 16:21, Rafael Weingärtner wrote:
> >>>>>>>> Thanks for your contribution Wido,
> >>>>>>>> I have not seen Rohit’s email; I will take a look
at it.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Ok, he has a FS here:
> >>>>>>>
> >>>>
> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Chimp
> >>>>>>>
> >>>>>>>> About database schema changes happening only in X.Y,
I also agree
> >>> with
> >>>>>> you
> >>>>>>>> (that is a convention we all could agree on, and such
as conding
> >> and
> >>>>>>>> release procedures we could have a wiki page for that).
However, I
> >>>>>> think we
> >>>>>>>> still might have scripts in versions X.Y.Z to add data
to a table
> >>> such
> >>>>>> as
> >>>>>>>> “guest_os_hypervisor”.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Yes, that is true. A bugfix could be a addition into the
database,
> >>> but
> >>>>>>> we have to prevent it as much as possible.
> >>>>>>>
> >>>>>>>> The point to manage such scripts is that, if we are
in version
> >> such
> >>> as
> >>>>>>>> 4.7.0 and a new script emerges in version 4.5.3, we
would have to
> >>>>>> decide to
> >>>>>>>> run or not to run it. I would rather not run them, since
if they
> >> add
> >>>>>>>> something to the code base; those changes should also
be applied
> >>> into
> >>>>>>>> master and as a consequence it will be available in
a future
> >> update.
> >>>>>>>>
> >>>>>>>
> >>>>>>> I understand, but this is where our release cycle becomes
the
> >>> problem.
> >>>>>>> It is because we release a X.Y.Z release we run into these
kind of
> >>>>>> problems.
> >>>>>>>
> >>>>>>> If we as a project simple do not release the .Z releases
we would
> >> be
> >>>>>>> fine as well ;)
> >>>>>>>
> >>>>>>> You can try to complicate things with technical things,
or if we
> >>>> release
> >>>>>>> every two / three weeks we don't run into these kind of
situations
> >> :)
> >>>>>>>
> >>>>>>> We might even cut the database version loose from the code
version.
> >>>>>>>
> >>>>>>> Database version is simple 100, 101, 102, 103, 104, 105.
And a code
> >>>>>>> version requires a certain version of the database.
> >>>>>>>
> >>>>>>> Wido
> >>>>>>>
> >>>>>>>> On Mon, Dec 28, 2015 at 12:50 PM, Wido den Hollander
<
> >>> wido@widodh.nl>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 28-12-15 14:16, Rafael Weingärtner wrote:
> >>>>>>>>>> Hi all devs,
> >>>>>>>>>> First of all, sorry the long text, but I hope
we can start a
> >>>>>> discussion
> >>>>>>>>>> here and improve that part of ACS.
> >>>>>>>>>>
> >>>>>>>>>> A while ago I have faced the code that Apache
CloudStack (ACS)
> >>> uses
> >>>> to
> >>>>>>>>>> upgrade from a version to newer one and that
did not seem to be
> >> a
> >>>> good
> >>>>>>>>> way
> >>>>>>>>>> to execute our upgrades. Therefore, I decided
to use some time
> >> to
> >>>>>> search
> >>>>>>>>>> for alternatives.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I think we all saw that happen once or more :)
> >>>>>>>>>
> >>>>>>>>>> I have read some material about versioning of
scripts used to
> >>>> upgrade
> >>>>>> a
> >>>>>>>>>> database (DB) of a system and went through some
frameworks that
> >>>> could
> >>>>>>>>> help
> >>>>>>>>>> us.
> >>>>>>>>>>
> >>>>>>>>>> In the literature of software engineering, it
is firmly stated
> >>> that
> >>>> we
> >>>>>>>>> have
> >>>>>>>>>> to version DB scripts as we do with the source
code of the
> >>>>>> application,
> >>>>>>>>>> using the baseline approach. Gladly, we were
not that bad at
> >> this
> >>>>>> point,
> >>>>>>>>> we
> >>>>>>>>>> already versioned our routines for DB upgrade
(.sql and .java).
> >>>>>>>>> Therefore,
> >>>>>>>>>> it seemed that we just did not have used a practical
approach to
> >>>> help
> >>>>>> us
> >>>>>>>>>> during DB upgrades.
> >>>>>>>>>>
> >>>>>>>>>> From my readings and looking at the ACS source
code I raised the
> >>>>>>>>> following
> >>>>>>>>>> requirement:
> >>>>>>>>>> • We should be able to write more than one
routine to upgrade
> >> to a
> >>>>>>>>>> version; those routines can be written in Java
and SQL. We might
> >>>> have
> >>>>>>>>> more
> >>>>>>>>>> than a routine to be executed for each version
and we should be
> >>> able
> >>>>>> to
> >>>>>>>>>> define an order of execution. Additionally,
to go to an upper
> >>>>>> version, we
> >>>>>>>>>> have to run all of the routines from smaller
versions first,
> >> until
> >>>> we
> >>>>>>>>>> achieve the desired version.
> >>>>>>>>>>
> >>>>>>>>>> We could also add another requirement that is
the downgrade
> >> from a
> >>>>>>>>> version,
> >>>>>>>>>> which we currently do not support. With that
comes my first
> >>> question
> >>>>>> for
> >>>>>>>>>> discussion:
> >>>>>>>>>> • Do we want/need a method to downgrade from
a version to a
> >>> previous
> >>>>>>>>> one?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I personally do not care. Usually people should
create a backup
> >>> PRIOR
> >>>>>> to
> >>>>>>>>> a upgrade. If that fails they can restore the backup.
> >>>>>>>>>
> >>>>>>>>>> I found an explanation for not supporting downgrades,
and I
> >> liked
> >>>> it:
> >>>>>>>>>> http://flywaydb.org/documentation/faq.html#downgrade
> >>>>>>>>>>
> >>>>>>>>>> So, what I devised for us:
> >>>>>>>>>> First the bureaucracy part - our migrations
occur basically in
> >>> three
> >>>>>> (3)
> >>>>>>>>>> steps, first we have a "prepare script", then
a cleanup script
> >> and
> >>>>>>>>> finally
> >>>>>>>>>> the migration per se that is written in Java,
at least, that is
> >>> what
> >>>>>> we
> >>>>>>>>> can
> >>>>>>>>>> expect when reading the interface
> >>> “com.cloud.upgrade.dao.DbUpgrade”.
> >>>>>>>>>>
> >>>>>>>>>> Additionally, our scripts have the following
naming convention:
> >>>>>>>>>> schema-<currentVersion>to<desiredVersion>,
which in IMHO may
> >> cause
> >>>>>> some
> >>>>>>>>>> confusion because at first sight we may think
that from the same
> >>>>>> version
> >>>>>>>>> we
> >>>>>>>>>> could have different paths to an upper version,
which in
> >> practice
> >>> is
> >>>>>> not
> >>>>>>>>>> happening. Instead of a <currentVersion>to<version>
we could
> >>> simply
> >>>>>> use
> >>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension>,
giving that,
> >> we
> >>>>>> have to
> >>>>>>>>>> execute all of the V_<version> scripts
that are smaller than the
> >>>>>> version
> >>>>>>>>> we
> >>>>>>>>>> want to upgrade.
> >>>>>>>>>>
> >>>>>>>>>> To clarify what I am saying, I will use an example.
Let’s say we
> >>>> have
> >>>>>>>>> just
> >>>>>>>>>> installed ACS and ran the cloudstack-setup-database.
That
> >> command
> >>>> will
> >>>>>>>>>> create a database schema in version 4.0.0. To
upgrade that
> >> schema
> >>> to
> >>>>>>>>>> version 4.3.0 (it is just an example, it could
be any other
> >>>> version),
> >>>>>> ACS
> >>>>>>>>>> will use the following mapping:
> >>>>>>>>>>
> >>>>>>>>>> _upgradeMap.put("4.0.0", new DbUpgrade[] {new
Upgrade40to41(),
> >> new
> >>>>>>>>>> Upgrade410to420(), new Upgrade420to421(), new
Upgrade421to430())
> >>>>>>>>>>
> >>>>>>>>>> After loading the mapping, ACS will execute
the scripts defined
> >> in
> >>>>>> each
> >>>>>>>>> one
> >>>>>>>>>> of the Upgrade path classes and the migration
code per se.
> >>>>>>>>>>
> >>>>>>>>>> Now, let’s say we change the “.sql” scripts
name to the pattern
> >> I
> >>>>>>>>>> mentioned, we would have the following scripts;
those are the
> >>>> scripts
> >>>>>>>>> found
> >>>>>>>>>> that aim to upgrade to versions between the
interval 4.0.0 –
> >> 4.3.0
> >>>>>>>>>> (considering 4.3.0, since that is the goal version):
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> - schema-40to410, can be named to: V_410_A.sql
> >>>>>>>>>> - schema-40to410-cleanup, can be named to: V_410_B.sql
> >>>>>>>>>> - schema-410to420, can be named to: V_420_A.sql
> >>>>>>>>>> - schema-410to420-cleanup , can be named to:
V_420_b.sql
> >>>>>>>>>> - schema-420to421, can be named to: V_421_A.sql
> >>>>>>>>>> - schema-421to430, can be named to: V_430_A.sql
> >>>>>>>>>> - schema-421to430-cleanup, can be named to:
V_430_B.sql
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Additionally, all of the java code would have
to follow the same
> >>>>>>>>>> convention. For instance, we have
> >>>>>> “com.cloud.upgrade.dao.Upgrade40to41”,
> >>>>>>>>>> which has some java code to migrate from 4.0.0
to 4.1.0. The
> >> idea
> >>> is
> >>>>>> to
> >>>>>>>>>> extract that migration code to a Java class
named: V_410_C.java,
> >>>>>> giving
> >>>>>>>>>> that it has to execute the SQL scripts before
the java code.
> >>>>>>>>>>
> >>>>>>>>>> In order to go from a smaller version (4.0.0)
to an upper one
> >>>>>> (4.3.0), we
> >>>>>>>>>> have to run all of the migration routines from
intermediate
> >>>> versions.
> >>>>>>>>> That
> >>>>>>>>>> is what we are already doing, but we do all
of that manually.
> >>>>>>>>>>
> >>>>>>>>>> Bottom line, I think we could simple use the
convention
> >>>>>>>>>> V_<numberOfVersion>_<sequencial>.<fileExtension>
to name upgrade
> >>>>>>>>> routines.
> >>>>>>>>>> That would facilitate us to use a framework
to help us with that
> >>>>>> process.
> >>>>>>>>>> Additionally, I believe that we should always
assume that to go
> >>>> from a
> >>>>>>>>>> smaller version to a higher one, we should run
all of the
> >> scripts
> >>>> that
> >>>>>>>>>> exist between them. What do you guys think of
that?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That seems good to me. But we still have to prevent
that we
> >> perform
> >>>>>>>>> database changes in a X.Y.Z release since that is
branched off
> >> to a
> >>>>>>>>> different branch.
> >>>>>>>>>
> >>>>>>>>> Imho database changes should only happen in X.Y
releases.
> >>>>>>>>>
> >>>>>>>>>> After the bureaucracy, we can discuss tools.
If we use that
> >>>>>> convention to
> >>>>>>>>>> name migration (upgrade) routines, we can start
thinking on
> >> tools
> >>> to
> >>>>>>>>>> support our migration process. I found two (2)
promising ones:
> >>>>>> Liquibase
> >>>>>>>>>> and Flywaydb (both seem to be under Apache license,
but the
> >> first
> >>>> one
> >>>>>> has
> >>>>>>>>>> an enterprise version?!). After reading the
documentation and
> >> some
> >>>>>> usage
> >>>>>>>>>> examples I found the flywaydb easier and simpler
to use.
> >>>>>>>>>>
> >>>>>>>>>> What are the options of tools that we can use
to help us manage
> >>> the
> >>>>>>>>>> database upgrade, without needing to code the
upgrade path that
> >>> you
> >>>>>> know?
> >>>>>>>>>>
> >>>>>>>>>> After that, I think we should decide if we should
create another
> >>>>>>>>>> project/component to take care of migrations,
or we can just add
> >>> the
> >>>>>>>>>> dependency of the tool to a project such as
“cloud-framework-db”
> >>> and
> >>>>>>>>> start
> >>>>>>>>>> using it.
> >>>>>>>>>>
> >>>>>>>>>> The “cloud-framework-db” project seems to
have a focus on other
> >>>> things
> >>>>>>>>> such
> >>>>>>>>>> as managing transactions and generating SQLs
from annotations
> >> (?!?
> >>>>>> That
> >>>>>>>>>> should be a topic for another discussion). Therefore,
I would
> >>> rather
> >>>>>>>>> create
> >>>>>>>>>> a new project that has the specific goal of
managing ACS DB
> >>>> upgrades.
> >>>>>> I
> >>>>>>>>>> would also move all of the routines (SQL and
Java) to this new
> >>>>>> project.
> >>>>>>>>>> This project would be a module of the CloudStack
project and it
> >>>> would
> >>>>>>>>>> execute the upgrade routines at the startup
of ACS.
> >>>>>>>>>>
> >>>>>>>>>> I believe that going from a homemade solution
to one that is
> >> more
> >>>>>>>>>> consolidated and used by other communities would
be the way to
> >> go.
> >>>>>>>>>>
> >>>>>>>>>> I can volunteer myself to create a PR with the
aforementioned
> >>>> changes
> >>>>>> and
> >>>>>>>>>> using flywaydb to manage our upgrades. However,
I prefer to
> >> have a
> >>>>>> good
> >>>>>>>>>> discussion with other devs first, before starting
coding.
> >>>>>>>>>>
> >>>>>>>>>> Do you have suggestions or points that should
be raised before
> >> we
> >>>>>> start
> >>>>>>>>>> working on that?
> >>>>>>>>>
> >>>>>>>>> Rohit suggested Chimp earlier this year:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201508.mbox/%3C677BD09F-FC75-4888-8DC8-2B7AF7439221@shapeblue.com%3E
> >>>>>>>>>
> >>>>>>>>> The thread is called: "[DISCUSS] Let's fix CloudStack
Upgrades
> >> and
> >>> DB
> >>>>>>>>> migrations with CloudStack Chimp"
> >>>>>>>>>
> >>>>>>>>> Maybe there is something good in there.
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Rafael Weingärtner
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>> Regards.
> >>>>>>
> >>>>>> Find out more about ShapeBlue and our range of CloudStack related
> >>>> services:
> >>>>>> IaaS Cloud Design & Build
> >>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
CSForge –
> >>> rapid
> >>>>>> IaaS deployment framework <http://shapeblue.com/csforge/>
> >>>>>> CloudStack Consulting <http://shapeblue.com/cloudstack-consultancy/
> >>>
> >>> |
> >>>> CloudStack
> >>>>>> Software Engineering
> >>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
> >>>>>> CloudStack Infrastructure Support
> >>>>>> <http://shapeblue.com/cloudstack-infrastructure-support/>
|
> >>> CloudStack
> >>>>>> Bootcamp Training Courses <
> >> http://shapeblue.com/cloudstack-training/>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Rafael Weingärtner
> >>>
> >>
> >>
> >>
> >> --
> >> Daan
> >>
> >
> >
> >
>



-- 
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message