incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Huang <Alex.Hu...@citrix.com>
Subject RE: [DISCUSS] Fixing cloud-setup-databases
Date Mon, 11 Feb 2013 18:15:10 GMT
Rohit,

The two should be one.  Having the two separate is a bad idea because it means developers
is running something very different from the actual deployment version which means actual
deployment version is not tested until QA hits it.  The setup database script should give
options and the target in maven should just have all of the options standardized for developer.

The other thing is you should announce this to the list with big headlines.  Starting 4.1,
we no longer modify the create-schema.sql.  Only modify the upgrade schemas.  The process
should look like this and should be documented.

- For every major release (first dot release), we need to roll-up all of the changes in the
schema and produce a create-schema.sql.  The version recorded inside the create-schema.sql
is the version of the major release.  The steps to do this should be recorded in a wiki for
the release manager to use.
- Every release from this major release, we always create the database from create-schema.sql
and then run the upgrades.  The upgrades are exactly the same as what the management server
would do to upgrade from existing version to current.

We should discuss the following things:
- Management Server no longer auto-upgrades.  It brings more problems than it solves.  It
should just detect that the version in the database is different than its software version
and exit.  We can provide a flag to override that function.  I just didn't think this through
when I designed it but the good thing is it's all self-contained in a plugin so it's not difficult
to change it.
- DatabaseCreator should take care of both upgrade situations and initial schema creation
(which I believe you did already) and upgrade cleanup (not sure if you did this part) after
all management servers have been upgraded.

This means 
- Developers don't write schemas in two places.  It's always written in the upgrade files.
- Upgrade process for the admin is multi-steps.
	- Run DatabaseCreator (may be in a script) to do pre-upgrade and as part of this step should
be the script to dump the database in case problems happen.
	- Upgrade all management server
	- Run DatabaseCreator (again maybe in a script) to do post-upgrade cleanup.
- Release manager produces the create-schema.sql for every major release.

For 4.1, we need to get rid of the 4.1-new-schema.  I have no idea why that's created as it
obviously doesn't fit into our upgrade strategy.  Everything in that file should be moved
into the DB upgrade scripts and that file removed in git and from the developer/pom.xml. 
Also, the version in create-schema.sql needs to stay the same and not changed with the release
version any more.

--Alex

> -----Original Message-----
> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com] On Behalf
> Of Rohit Yadav
> Sent: Monday, February 11, 2013 9:13 AM
> To: Hugo Trippaers
> Cc: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
> 
> Forgot to mention, one more reason why it was written in Java, as we
> pass a database upgrade class which is suppose to do the upgrade,
> tests etc. If we take the python route, these things can be python
> modules instead of java classes to do the magic.
> 
> Regards.
> 
> On Mon, Feb 11, 2013 at 10:38 PM, Rohit Yadav <bhaisaab@apache.org>
> wrote:
> > Initially I thought to write it in python, which I can do it now as
> > well. I just wanted to re-use some of the stuff that Transaction and
> > ScriptRunner provides. Just let me know if it's a pain to package, we
> > don't need the java one, besides it's really small and I can write a
> > replacement in python (because it's available on most linux distros
> > and osx, can write in any other language as well) in no time.
> >
> > Regards.
> >
> > On Mon, Feb 11, 2013 at 10:32 PM, Hugo Trippaers
> > <HTrippaers@schubergphilis.com> wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com] On
> Behalf
> >>> Of Rohit Yadav
> >>> Sent: Monday, February 11, 2013 5:49 PM
> >>> To: Hugo Trippaers
> >>> Cc: cloudstack-dev@incubator.apache.org
> >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
> >>>
> >>> On Mon, Feb 11, 2013 at 9:57 PM, Hugo Trippaers
> >>> <HTrippaers@schubergphilis.com> wrote:
> >>> > Hey Rohit,
> >>> >
> >>> > As far as I know these are two disctict use cases.
> >>> >
> >>> I agree, but only in few sets of sql files. The reasons why
> DatabaseCreator
> >>> was introduced:
> >>>
> >>> 0. Make life of sysadmins and developers easier 1. If passed --database,
> it
> >>> would drop and recreate databases 2. Utility to run sql files, based on
> passwd
> >>> db.properties file (no hardcoded db.props file) 3. Run db upgrader class,
> so
> >>> we won't have to run custom upgrade path scripts
> >>>
> >>> > The maven deploy db is just for developers or at least people who are
> just
> >>> working from source directly, and should provide a convenient way for
> >>> developers to refresh their test database or deploy a new schema.
> >>>
> >>> So, maven's deploydb and cloud-setup-databases (or cloudstack-setup-
> >>> databases now) take different routes. Channeling both of them through
> >>> DatabaseCreator would ensure same kind of database
> deploying/creating
> >>> environment for both developers and sysadmins, less bugs and issues.
> >>>
> >>> It's not required that you've to pass the same set of sql files or
> configuration
> >>> in both cases, for example the developer-prefill or devcloud.sql should
> not
> >>> be part of cloudstack-setup-databases.
> >>>
> >>> I saw three files which were not part of one of the use cases, so I asked
> why
> >>> they are there and who's using them. I don't know so I ask
> >>> :)
> >>>
> >>> > The cloud-setup-databases is a tool that is used the sysadmins to setup
> a
> >>> database after they have deployed the packages. It has several
> purposes,
> >>> setup db.properties, configure encryption and load a database (if the
> >>> supplied database is empty).
> >>>
> >>> Yes, my point is as a developer I want to test/play around them as well.
> >>>
> >>> >
> >>> > In my view both should remain for these purposes, but they should be
> in-
> >>> sync and also the upgrade sql script should be in sync with these changes.
> >>>
> >>> You got it! Keeping them in sync is an issue, channeling them through
> >>> DatabaseCreator would make things little more maintainable. I also see
> >>> DatabaseCreator as a utility that can be perhaps used by plugins and
> external
> >>> tools as well (distant future) to do magical stuff, reusable and
> maintainable.
> >>
> >> Sounds good to me :-)  Anything that makes my life as an admin easier
> gets a +1. Downside it that it is another java tool that has requirements and
> dependencies and needs to be packaged to be use full. We are already
> sneaking in some jasypt jars into /usr/share/java to not break the existing
> scripts. Having a separate java bases tool would probably require its own set
> of dependencies.
> >>
> >>>
> >>> Thoughts, comments?
> >>>
> >>> Regards.
> >>>
> >>> >
> >>> > Cheers,
> >>> >
> >>> > Hugo
> >>> >
> >>> >> -----Original Message-----
> >>> >> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com] On
> >>> >> Behalf Of Rohit Yadav
> >>> >> Sent: Monday, February 11, 2013 11:15 AM
> >>> >> To: cloudstack-dev@incubator.apache.org
> >>> >> Subject: [DISCUSS] Fixing cloud-setup-databases
> >>> >>
> >>> >> DatabaseCreator was introduce for 4.1 so both maven deploydb
> target
> >>> >> and cloud-setup-databases would use that.
> >>> >> Following does not run on maven's deploydb:
> >>> >> cloudbridge_db.sql
> >>> >> schema-level.sql
> >>> >> server-setup.sql
> >>> >>
> >>> >> Following runs only from maven's deploydb:
> >>> >> com.cloud.upgrade.DatabaseUpgradeChecker
> >>> >>
> >>> >> Following runs only from cloud-setup-databases:
> >>> >> com.cloud.test.DatabaseConfig
> >>> >>
> >>> >> Pl. advise if we need to run those as part of maven's deploydb,
or
> >>> >> cloud- setup-database or both?
> >>> >>
> >>> >> Regards.

Mime
View raw message