Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of donal.lafferty@citrix.com
 designates 185.25.65.24 as permitted sender)
From: Donal Lafferty <donal.lafferty@citrix.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
CC: Daan Hoogland <DHoogland@schubergphilis.com>
Subject: RE: [DISCUSS] Checking in code that will break others' environments
Thread-Topic: [DISCUSS] Checking in code that will break others' environments
Thread-Index: 
 AQHPOLBrNj5QIBXqZEKpJ57teEV5lJrS4LqAgAAC5QCAAJ1FAIAAA0wAgAAUH4CAACangIAAAo4AgABBaAD///A4gIAAEVxA
Date: Thu, 6 Mar 2014 13:02:46 +0000
Message-ID: <9ADDE3F979256644BED8F0D244BE51F00FA832@AMSPEX01CL02.citrite.net>
References: 
 <CACOnxChoHbXa75ZAD_bdPXRgeJrxospyZJQWLAJDkuKM7GmSVw@mail.gmail.com>
 <CAEJ3w4XWywoxVeP9GPAQdxQrWuY2xgtvh36xxULr2dKQCepOcQ@mail.gmail.com>
 <CACOnxCg5d5rgNqdub78btJOjk8KXxoAaDDs1M+brjOBogyQodw@mail.gmail.com>
 <697ECC0B-FEEB-46E8-84E3-FDA190615A5B@citrix.com>
 <CACOnxCiELULECvQR38QOpepHZ-7g9NhfzXt_35gK+NCy=5snVQ@mail.gmail.com>
 <DCF3A557-D18B-4E53-BA91-3AFD88ADE3FC@citrix.com>
 <02E3616756CDC04687D08374DA63D32C4C083B@SBPOMB401.sbp.lan>
 <CAGQtxvan9JasowCBrwP20LAp=qhODioo1xCZ8X1bkwC7ZtqVmg@mail.gmail.com>
 <9ADDE3F979256644BED8F0D244BE51F00FA7FC@AMSPEX01CL02.citrite.net>
 <CAGQtxvZag1jHJwFk5WjtSiaZv13mz+fodgULwc-CU4FgWUt=jg@mail.gmail.com>
In-Reply-To: 
 <CAGQtxvZag1jHJwFk5WjtSiaZv13mz+fodgULwc-CU4FgWUt=jg@mail.gmail.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The thread started with a discussion of problems with out of date databases=
.  Mike pulled from master, rebuilt, and found out his testbed's database w=
as out of date and no longer works.

We've tried to find a database solution, but out of databases are caused by=
 more than schema changes.  The data in the tables and other pieces of Clou=
dStack can change after a commit, especially towards the end of release whe=
n everyone's feature is being merged.

With 4.2, I dealt with this issue by writing a script that rebuilt my testb=
ed by replaying the API calls I made to set it up in the first place.

Is this a solution you've considered for your dev environment?=20

DL
=20
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: 06 March 2014 12:50
> To: dev
> Cc: Daan Hoogland
> Subject: Re: [DISCUSS] Checking in code that will break others' environme=
nts
>=20
> No, but how do you mean with respect to this thread?
>=20
> On Thu, Mar 6, 2014 at 1:48 PM, Donal Lafferty <donal.lafferty@citrix.com=
>
> wrote:
> > Hi Daan,
> >
> > Is there anything stopping you from scripting the configuration of
> > your CloudStack testbed?  E.g. with Marvin or CloudMonkey
> >
> > DL
> >
> >> -----Original Message-----
> >> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> >> Sent: 06 March 2014 09:53
> >> To: dev
> >> Cc: Daan Hoogland
> >> Subject: Re: [DISCUSS] Checking in code that will break others'
> >> environments
> >>
> >> I totally agree with the incremental approach. I am a fascist at time
> >> because i would even want people to add downgrade scripts to any db
> >> change they make. Having them not adjust their sql is a good first ste=
p,
> though.
> >>
> >> On Thu, Mar 6, 2014 at 10:43 AM, Miguel Ferreira
> >> <MFerreira@schubergphilis.com> wrote:
> >> > Hi all,
> >> >
> >> > I've been looking at the way DB related code (SQL scripts and Java
> >> > classes)
> >> are updated and what is the impact of those updates on a live cloudsta=
ck
> DB.
> >> > By the way, my intention is to support a per-commit DB upgrade of a
> >> running system.
> >> >
> >> > Anyway, I completely agree that sending an email warning people
> >> > about
> >> changes that can potentially break running environments is a good
> >> thing, and can save a lot of time and headaches.
> >> > I also like the idea of Rajani of using tools to help in the DB
> >> > migration
> >> process.
> >> > However, I would like to put forward another idea: what about
> >> > making sure
> >> that the changes to the DB (schema and data) are always incremental?
> >> >
> >> > I mean, once a SQL statement (say A) is added to a script it could
> >> > be kept as
> >> is, and subsequent modifications cloud be made via new statements
> >> (say B), instead of adapting A.
> >> > This would allow people to upgrade their databases to the point
> >> > they ran
> >> statement A, and afterwards upgrade it again to the point they ran
> >> statement B.
> >> > The same principle could also be applied to the Java classes that
> >> > do data
> >> migration, but maybe there it might be a bit more involved.
> >> >
> >> > Cheers,
> >> > Miguel
> >> >
> >> > -----Original Message-----
> >> > From: Koushik Das [mailto:koushik.das@citrix.com]
> >> > Sent: donderdag 6 maart 2014 8:25
> >> > To: <dev@cloudstack.apache.org>
> >> > Subject: Re: [DISCUSS] Checking in code that will break others'
> >> > environments
> >> >
> >> > Before doing a git pull, I generally check the sql schema changes
> >> > and run
> >> the delta manually on my existing setup. In most of the cases that
> >> works for me without having to redeploy the db.
> >> >
> >> > -Koushik
> >> >
> >> > On 06-Mar-2014, at 11:43 AM, Mike Tutkowski
> >> <mike.tutkowski@solidfire.com> wrote:
> >> >
> >> >> Yeah, I definitely just meant a "heads up" during development if
> >> >> you are going to change something that will break other people's
> >> >> environments who update. If these people know in advance, they may
> >> >> choose to postpone an update until they are at a better point.
> >> >>
> >> >>
> >> >> On Wed, Mar 5, 2014 at 11:01 PM, Rajani Karuturi
> >> >> <Rajani.Karuturi@citrix.com
> >> >>> wrote:
> >> >>
> >> >>> Across versions db migration is taken care. I think this is bound
> >> >>> to occur while working on a release, if multiple people work on
> >> >>> the same branch with different work-in-progress features.
> >> >>>
> >> >>> Could we move to flyway or liquibase which can take care of db
> >> >>> versioning and migration?
> >> >>>
> >> >>>
> >> >>> ~Rajani
> >> >>>
> >> >>>
> >> >>>
> >> >>> On 06-Mar-2014, at 2:08 am, Mike Tutkowski
> >> >>> <mike.tutkowski@solidfire.com>
> >> >>> wrote:
> >> >>>
> >> >>>> Yeah, in this case, I'm not referring to erroneous code that
> >> >>>> breaks a person's environment (since hopefully the person
> >> >>>> wouldn't have knowingly checked in such code), but rather, say,
> >> >>>> DB-type changes that improve the system, but break current
> setups.
> >> >>>>
> >> >>>> Just a heads-up e-mail with some easily identifiable tag.
> >> >>>>
> >> >>>> Can anyone think of a good tag for this? It's not always DB
> >> >>>> related, so
> >> >>> we
> >> >>>> might want the tag to be more general.
> >> >>>>
> >> >>>>
> >> >>>> On Wed, Mar 5, 2014 at 1:28 PM, Ian Duffy <ian@ianduffy.ie>
> wrote:
> >> >>>>
> >> >>>>> +1 to this.
> >> >>>>>
> >> >>>>> Having the build suddenly break due to a git pull has been very
> >> >>> annoying!
> >> >>>>> I usually end up searching through the commit log and doing a
> >> >>>>> resets until I find a commit where it works. Then waiting
> >> >>>>> awhile until I do a git pull again and hoping the code was fixed=
.
> >> >>>>>
> >> >>>>> On 5 March 2014 20:19, Mike Tutkowski
> >> >>>>> <mike.tutkowski@solidfire.com>
> >> >>>>> wrote:
> >> >>>>>> Hi,
> >> >>>>>>
> >> >>>>>> I encountered a bit of a problem this morning and thought I
> >> >>>>>> would bring
> >> >>>>> it
> >> >>>>>> up for discussion.
> >> >>>>>>
> >> >>>>>> If we already have a policy around this, please let me know.
> >> >>>>>>
> >> >>>>>> So, I fetched the latest and rebased my local 4.4 development
> >> >>>>>> branch on
> >> >>>>> top
> >> >>>>>> of master. This all went just fine.
> >> >>>>>>
> >> >>>>>> When I rebuilt and re-started the CS Management Server, I soon
> >> >>> realized I
> >> >>>>>> could no longer log in from the GUI.
> >> >>>>>>
> >> >>>>>> As it turns out, the DB schema had been updated and so my
> >> >>>>>> database was
> >> >>>>> out
> >> >>>>>> of date. The code was querying for fields that didn't exist in =
my
> DB.
> >> >>>>>>
> >> >>>>>> As far as I know, the easiest way to get around this is to
> >> >>>>>> destroy my current cloud, run the script to re-build my
> >> >>>>>> database, then re-create
> >> >>> my
> >> >>>>>> cloud, which is somewhat time consuming.
> >> >>>>>>
> >> >>>>>> Do we have a process in place currently in which we ask those
> >> >>>>>> who make
> >> >>>>> such
> >> >>>>>> changes to send out a notification e-mail to dev@ to give
> >> >>>>>> people a
> >> >>>>> heads up
> >> >>>>>> that updating will lead to such issues? On previous projects,
> >> >>>>>> we would
> >> >>>>> send
> >> >>>>>> out an e-mail and then people could be aware to only update if
> >> >>>>>> they
> >> >>> were
> >> >>>>>> prepared for such re-work.
> >> >>>>>>
> >> >>>>>> To be clear here, I'm not meaning to pick on anyone in
> >> >>> particular...this
> >> >>>>>> has happened several times over the course of my CloudStack
> >> >>>>>> development
> >> >>>>> and
> >> >>>>>> I expect that I, too, have checked in such code (without
> >> >>>>>> sending out a relevant e-mail) that lead people to have to
> >> >>>>>> perform such a complete re-build action un-expectedly.
> >> >>>>>>
> >> >>>>>> What do people think about this? Maybe we should just add an
> >> >>>>>> e-mail tag
> >> >>>>> or
> >> >>>>>> something and point people to the relevant commit?
> >> >>>>>>
> >> >>>>>> Thanks!
> >> >>>>>>
> >> >>>>>> --
> >> >>>>>> *Mike Tutkowski*
> >> >>>>>> *Senior CloudStack Developer, SolidFire Inc.*
> >> >>>>>> e: mike.tutkowski@solidfire.com
> >> >>>>>> o: 303.746.7302
> >> >>>>>> Advancing the way the world uses the
> >> >>>>>> cloud<http://solidfire.com/solution/overview/?video=3Dplay>
> >> >>>>>> *(tm)*
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> *Mike Tutkowski*
> >> >>>> *Senior CloudStack Developer, SolidFire Inc.*
> >> >>>> e: mike.tutkowski@solidfire.com
> >> >>>> o: 303.746.7302
> >> >>>> Advancing the way the world uses the
> >> >>>> cloud<http://solidfire.com/solution/overview/?video=3Dplay>
> >> >>>> *(tm)*
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> *Mike Tutkowski*
> >> >> *Senior CloudStack Developer, SolidFire Inc.*
> >> >> e: mike.tutkowski@solidfire.com
> >> >> o: 303.746.7302
> >> >> Advancing the way the world uses the
> >> >> cloud<http://solidfire.com/solution/overview/?video=3Dplay>
> >> >> *(tm)*
> >> >
> >>
> >>
> >>
> >> --
> >> Daan
>=20
>=20
>=20
> --
> Daan