incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohit Yadav <bhais...@apache.org>
Subject Re: [DISCUSS] Fixing cloud-setup-databases
Date Wed, 13 Feb 2013 09:25:27 GMT
Unassigned CLOUDSTACK-1019 to be picked up by someone else.
I'm not sure how to do it correctly, picked up other things to work on.

TODO includes from the discussion on this thread:
- Fix cloudstack-setup-databases to use DatabaseCreator and to do that
we require configuring packaging so a classpath is expanded and used
to call DatabaseCreator
- Refactor and move out logic that does upgrades from mgmt server to
databasecreator.

Regards.

On Tue, Feb 12, 2013 at 5:02 PM, Abhinandan Prateek
<Abhinandan.Prateek@citrix.com> wrote:
> We can discuss this one ?
>
> On 12/02/13 3:46 PM, "Rohit Yadav" <bhaisaab@apache.org> wrote:
>
>>On Mon, Feb 11, 2013 at 11:45 PM, Alex Huang <Alex.Huang@citrix.com>
>>wrote:
>>> Rohit,
>>>
>>> The two should be one.  Having the two separate is a bad idea because
>>>it means developers is running something very different from the actual
>>>deployment version which means actual deployment version is not tested
>>>until QA hits it.  The setup database script should give options and the
>>>target in maven should just have all of the options standardized for
>>>developer.
>>>
>>> The other thing is you should announce this to the list with big
>>>headlines.  Starting 4.1, we no longer modify the create-schema.sql.
>>>Only modify the upgrade schemas.  The process should look like this and
>>>should be documented.
>>
>>+1
>>
>>> - For every major release (first dot release), we need to roll-up all
>>>of the changes in the schema and produce a create-schema.sql.  The
>>>version recorded inside the create-schema.sql is the version of the
>>>major release.  The steps to do this should be recorded in a wiki for
>>>the release manager to use.
>>
>>Okay will start a wiki on this. I just want to say that I don't know
>>how to fix it, I don't understand the requirements clearly but will
>>try my best to fix it.
>>
>>> - Every release from this major release, we always create the database
>>>from create-schema.sql and then run the upgrades.  The upgrades are
>>>exactly the same as what the management server would do to upgrade from
>>>existing version to current.
>>>
>>> We should discuss the following things:
>>> - Management Server no longer auto-upgrades.  It brings more problems
>>>than it solves.
>>
>>Alright, so there are a lot of Upgrade versionx-versiony classes which
>>are used by DatabaseUpgradeChecker. I did not write DatabaseCreator in
>>python because I wanted to skip rewriting logic for these upgrade
>>paths again. Do we want to keep it that way or should I start porting
>>them to Python?
>>
>>>  It should just detect that the version in the database is different
>>>than its software version and exit.  We can provide a flag to override
>>>that function.  I just didn't think this through when I designed it but
>>>the good thing is it's all self-contained in a plugin so it's not
>>>difficult to change it.
>>
>>Instead of management server calling DatabaseUpgradeChecker,
>>DatabaseCreator should call it. The upgrade path does not work in
>>DatabaseCreator because I'll have to clean the spring stuff and get
>>upgrading logic out of mgmt server if we don't want mgmt server to do
>>it.
>>
>>> - DatabaseCreator should take care of both upgrade situations and
>>>initial schema creation (which I believe you did already) and upgrade
>>>cleanup (not sure if you did this part) after all management servers
>>>have been upgraded.
>>>
>>
>>Yes, the initial schema creation work and no the upgrade and cleanup
>>does not work because they need some refactoring and mgmt server in
>>that case does not need to call DatabaseUpgradeChecker.
>>
>>> This means
>>> - Developers don't write schemas in two places.  It's always written in
>>>the upgrade files.
>>
>>+1
>>
>>> - Upgrade process for the admin is multi-steps.
>>
>>So, do we require sysadmin to run DatabaseCreator from version to
>>version or should n't it just automatically upgrade to latest version?
>>
>>>         - Run DatabaseCreator (may be in a script) to do pre-upgrade
>>>and as part of this step should be the script to dump the database in
>>>case problems happen.
>>>         - Upgrade all management server
>>>         - Run DatabaseCreator (again maybe in a script) to do
>>>post-upgrade cleanup.
>>> - Release manager produces the create-schema.sql for every major
>>>release.
>>>
>>> For 4.1, we need to get rid of the 4.1-new-schema.  I have no idea why
>>>that's created as it obviously doesn't fit into our upgrade strategy.
>>>Everything in that file should be moved into the DB upgrade scripts and
>>>that file removed in git and from the developer/pom.xml.  Also, the
>>>version in create-schema.sql needs to stay the same and not changed with
>>>the release version any more.
>>
>>Alright.
>>
>>Regards.
>>
>>>
>>> --Alex
>>>
>>>> -----Original Message-----
>>>> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com] On Behalf
>>>> Of Rohit Yadav
>>>> Sent: Monday, February 11, 2013 9:13 AM
>>>> To: Hugo Trippaers
>>>> Cc: cloudstack-dev@incubator.apache.org
>>>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
>>>>
>>>> Forgot to mention, one more reason why it was written in Java, as we
>>>> pass a database upgrade class which is suppose to do the upgrade,
>>>> tests etc. If we take the python route, these things can be python
>>>> modules instead of java classes to do the magic.
>>>>
>>>> Regards.
>>>>
>>>> On Mon, Feb 11, 2013 at 10:38 PM, Rohit Yadav <bhaisaab@apache.org>
>>>> wrote:
>>>> > Initially I thought to write it in python, which I can do it now as
>>>> > well. I just wanted to re-use some of the stuff that Transaction and
>>>> > ScriptRunner provides. Just let me know if it's a pain to package, we
>>>> > don't need the java one, besides it's really small and I can write a
>>>> > replacement in python (because it's available on most linux distros
>>>> > and osx, can write in any other language as well) in no time.
>>>> >
>>>> > Regards.
>>>> >
>>>> > On Mon, Feb 11, 2013 at 10:32 PM, Hugo Trippaers
>>>> > <HTrippaers@schubergphilis.com> wrote:
>>>> >>
>>>> >>
>>>> >>> -----Original Message-----
>>>> >>> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com]
On
>>>> Behalf
>>>> >>> Of Rohit Yadav
>>>> >>> Sent: Monday, February 11, 2013 5:49 PM
>>>> >>> To: Hugo Trippaers
>>>> >>> Cc: cloudstack-dev@incubator.apache.org
>>>> >>> Subject: Re: [DISCUSS] Fixing cloud-setup-databases
>>>> >>>
>>>> >>> On Mon, Feb 11, 2013 at 9:57 PM, Hugo Trippaers
>>>> >>> <HTrippaers@schubergphilis.com> wrote:
>>>> >>> > Hey Rohit,
>>>> >>> >
>>>> >>> > As far as I know these are two disctict use cases.
>>>> >>> >
>>>> >>> I agree, but only in few sets of sql files. The reasons why
>>>> DatabaseCreator
>>>> >>> was introduced:
>>>> >>>
>>>> >>> 0. Make life of sysadmins and developers easier 1. If passed
>>>>--database,
>>>> it
>>>> >>> would drop and recreate databases 2. Utility to run sql files,
>>>>based on
>>>> passwd
>>>> >>> db.properties file (no hardcoded db.props file) 3. Run db upgrader
>>>>class,
>>>> so
>>>> >>> we won't have to run custom upgrade path scripts
>>>> >>>
>>>> >>> > The maven deploy db is just for developers or at least
people
>>>>who are
>>>> just
>>>> >>> working from source directly, and should provide a convenient
way
>>>>for
>>>> >>> developers to refresh their test database or deploy a new schema.
>>>> >>>
>>>> >>> So, maven's deploydb and cloud-setup-databases (or
>>>>cloudstack-setup-
>>>> >>> databases now) take different routes. Channeling both of them
>>>>through
>>>> >>> DatabaseCreator would ensure same kind of database
>>>> deploying/creating
>>>> >>> environment for both developers and sysadmins, less bugs and
>>>>issues.
>>>> >>>
>>>> >>> It's not required that you've to pass the same set of sql files
or
>>>> configuration
>>>> >>> in both cases, for example the developer-prefill or devcloud.sql
>>>>should
>>>> not
>>>> >>> be part of cloudstack-setup-databases.
>>>> >>>
>>>> >>> I saw three files which were not part of one of the use cases,
so
>>>>I asked
>>>> why
>>>> >>> they are there and who's using them. I don't know so I ask
>>>> >>> :)
>>>> >>>
>>>> >>> > The cloud-setup-databases is a tool that is used the sysadmins
>>>>to setup
>>>> a
>>>> >>> database after they have deployed the packages. It has several
>>>> purposes,
>>>> >>> setup db.properties, configure encryption and load a database
(if
>>>>the
>>>> >>> supplied database is empty).
>>>> >>>
>>>> >>> Yes, my point is as a developer I want to test/play around them
as
>>>>well.
>>>> >>>
>>>> >>> >
>>>> >>> > In my view both should remain for these purposes, but they
>>>>should be
>>>> in-
>>>> >>> sync and also the upgrade sql script should be in sync with
these
>>>>changes.
>>>> >>>
>>>> >>> You got it! Keeping them in sync is an issue, channeling them
>>>>through
>>>> >>> DatabaseCreator would make things little more maintainable.
I also
>>>>see
>>>> >>> DatabaseCreator as a utility that can be perhaps used by plugins
>>>>and
>>>> external
>>>> >>> tools as well (distant future) to do magical stuff, reusable
and
>>>> maintainable.
>>>> >>
>>>> >> Sounds good to me :-)  Anything that makes my life as an admin
>>>>easier
>>>> gets a +1. Downside it that it is another java tool that has
>>>>requirements and
>>>> dependencies and needs to be packaged to be use full. We are already
>>>> sneaking in some jasypt jars into /usr/share/java to not break the
>>>>existing
>>>> scripts. Having a separate java bases tool would probably require its
>>>>own set
>>>> of dependencies.
>>>> >>
>>>> >>>
>>>> >>> Thoughts, comments?
>>>> >>>
>>>> >>> Regards.
>>>> >>>
>>>> >>> >
>>>> >>> > Cheers,
>>>> >>> >
>>>> >>> > Hugo
>>>> >>> >
>>>> >>> >> -----Original Message-----
>>>> >>> >> From: rohityadav89@gmail.com [mailto:rohityadav89@gmail.com]
On
>>>> >>> >> Behalf Of Rohit Yadav
>>>> >>> >> Sent: Monday, February 11, 2013 11:15 AM
>>>> >>> >> To: cloudstack-dev@incubator.apache.org
>>>> >>> >> Subject: [DISCUSS] Fixing cloud-setup-databases
>>>> >>> >>
>>>> >>> >> DatabaseCreator was introduce for 4.1 so both maven
deploydb
>>>> target
>>>> >>> >> and cloud-setup-databases would use that.
>>>> >>> >> Following does not run on maven's deploydb:
>>>> >>> >> cloudbridge_db.sql
>>>> >>> >> schema-level.sql
>>>> >>> >> server-setup.sql
>>>> >>> >>
>>>> >>> >> Following runs only from maven's deploydb:
>>>> >>> >> com.cloud.upgrade.DatabaseUpgradeChecker
>>>> >>> >>
>>>> >>> >> Following runs only from cloud-setup-databases:
>>>> >>> >> com.cloud.test.DatabaseConfig
>>>> >>> >>
>>>> >>> >> Pl. advise if we need to run those as part of maven's
deploydb,
>>>>or
>>>> >>> >> cloud- setup-database or both?
>>>> >>> >>
>>>> >>> >> Regards.
>

Mime
View raw message