cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Animesh Chaturvedi <animesh.chaturv...@citrix.com>
Subject RE: [DISCUSS] Breaking out Marvin from CloudStack
Date Sun, 06 Oct 2013 18:10:50 GMT


> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Saturday, October 05, 2013 3:22 AM
> To: dev
> Subject: Re: [DISCUSS] Breaking out Marvin from CloudStack
> 
> H Chip,
> 
> As a feature-dev driven by a 150-man-strong-cloud-operator-base that is
> not interested in anything but me showing that they can work with what
> they asked for:
> 
> On Fri, Oct 4, 2013 at 10:52 PM, Chip Childers
> <chip.childers@sungard.com>wrote:
> ...
> 
> > On Fri, Oct 04, 2013 at 05:57:58PM +0530, Prasanna Santhanam wrote:
> > > I'll summarize and address the concerns raised so far.
> > >
> > > Marvin has been in this repo for a long time for us to start writing
> > > tests. The only tests I've seen coming are from a specific set of
> > > people focussed on QA efforts.
> >
> > I agree - and that's a problem.  New features should *ALL* have tests
> > before they merge into master.  I think that assuming that the only
> > test writers are a group of folks that write the tests today is
> > actually a larger problem.
> >
> 
> Yes and we will need to work down a backlog of scenarios before we ever
> can rely on guys like me doing that. Not because they won't but because
> there is to much to write tests for edging on the new features they
> write. Just because those tests aren't there yet. I think giving Citrix
> QA a repo to work on is fine but I would like to see it merged back at
> some point and a continued possibility to write them in the main tree.
> 
[Animesh>] While I don't agree to a separate repo for tests (marvin framework is ok) I
do want to call out the proposal is not for giving Citrix QA a repo to work on and I don't
think Prasanna meant that way. 

> >
> > > I want to reduce the impediment for
> > > people who are writing tests *today*. Those looking to get started
> > > in the near future won't have any new learning to do, just that
> > > their code goes in an alternate repo that is pointed to right
> > > infrastructure.
> >
> @prasanna: education of the developers is also something to consider. I
> know I could contribute more to integration and at Schuberg Philis we
> have some work going on in that direction. It would be nice to see easy
> integration of this with the Citrix work.
> 
> 
> >
> ...
> 
> > >
> > > Reason for tests and framework to go together is simple.
> >
> Guys, why not both. some general tests that will be valid for any ACS
> version can go in the marvin code base. Others that change over time
> should be tied to the version of ACS they apply to.
> 
> Test can be moved from one repo to the other if needed.
> 
> ...
> 
> >
> > That's marvin, not the tests, right?  If I add a new feature for 4.3,
> > should that test appear in a 4.2 run?  Aren't we aiming for (I know
> > we're not there) a situation where *all* tests pass before we release?
> > IMO not versioning the tests (not the Marvin framework) with the
> > target of the tests is confusing.
> >
> That I agree with completely.
> 
> 
> > > The infrastructure (currently) only looks at the changes in the test
> > > directory before performing a run. It doesn't care whether server/
> > > was changed or plugins/x/y/z was changed. That's because the tests
> > > are unrelated to what is in the rest of the repository. In fact you
> > > can't even run them without a deployed cloud. So I don't see why
> > > idle code should lie in the repo.
> >
> You can trigger the building of a cloud when code changes and run the
> tests against that.
> 
> >  >
> > > Integration tests are essential, they will keep coming as long as
> > > Citrix QA is invested in the effort, but they need to come faster
> > > into the repos and that will be addressed by the separation IMO.
> >
> > How?
> >
> > > Managing
> > > the feature submitted to cloudstack against tests submitted to
> > > marvin is not a hard thing to do. We simply mirror the release
> > > branches in marvin and submit tests there. In fact I wonder why we
> > > didn't have this question when docs were separated? It doesn't work
> > > any differently really.
> >
> > Well, it turns out that docs are absolutely not being done before code
> > comes into master.  I dislike this fact, but live with it.
> >
> I don't see why a feature has to be published when it is implemented. So
> I don't agree on the docs. The great advantage of hidden features is
> that you can take them out again more easily when they don't meet the
> promise they made when implemented. So Docs shouldn't be written (or at
> least published) until features are used in real live imho.
> 
> 
> > > What I would like to see provided by CloudStack is the ability to
> > > upgrade all our test environments, staging environments, UATs, what
> > > have you in continuous integration and have tests run on the
> > > upgraded setup. That allows incrementally testing CloudStack the way
> > > users do it. The current design of installing everything from
> > > scratch, redoing the testbed for each automated test run is mostly a
> > > workaround for that inability. If we had this ability tests written
> > > in marvin can be run against live setups at all times as and when
> > > features merge to master.
> >
> Yes and it would still be great if we could build these installations of
> arbitrary version as required to de upgrades from any version to any
> higher version and run the tests on those. For that marvin (or some tool
> using
> marvin) must be able to preinstall a setup on an version x upgrade to y
> and then run the required test.
> 
> > It has always been my goal to reduce the barrier to writing tests for
> > > cloudstack and that's entirely the goal of the refactor as well. So
> > > I hope I have spent my time well. Since I can't be after people to
> > > write tests, and setup mandates, I aim to address the technical
> > > difficulty in writing tests through this refactor and separation.
> >
> 
> But you do agree that making all of us write tests, unit as well as
> integration, is the ultimate goal, don't you.
> 
> 
> > I'll say one thing though...  I'm exceptionally frustrated with the
> > features / commits coming into master with little to no tests (unit OR
> > integration).  I'm at a loss at this point, and frankly considering a
> > possible reality that this community (generally) just doesn't care
> > about making it easy to reproduce tests.  The lack of interest and
> > ignorance about the pain that this causes is infuriating to me.
> >
> Here I can't speak for all of us featurists but I want my features to
> work after I drop them. And I have the good sense to see that unit tests
> are not sufficient for that. I feel guilt as well as infuriation. Taking
> a side here against my wish: It is hard to learn 2 million lines of code
> implement a feature see it work on a version and then try to guarantee
> to the rest of the world that it will not break a twig down under. We
> should. We should also make this as easy as we can. So for now we
> shouldn't ever accept a submission without unit tests and strive to work
> towards a situation where we don't have to accept submissions without
> integration tests any longer.
> 
> Let's be practical about it and make sure that both paths (marvin and
> cloudstack repo) are open and stimulate the use of both.
> 
> One other thing: I refer to my self as the guilty party somehow. This is
> nonsense but it helps me focus on the goal. One goal is to have people
> outside Citrix write integration tests. I wrote integration tests for
> some features that where a rip-off of some integration tests that Hugo
> wrote for his. We run them once in a while (as in every release but not
> every RC) I am yet to learn the Citrix way of doing this and will focus
> on that. You can not blame a featurist like my self to first focus on
> the code to get something done. Now that o have a grasp of a few percent
> of those two million lines I am happy to learn more about marvin and
> rewrite our tests.
> 
> patients with me please, I am willing,
> Daan, (and I am sure loads of others)

Mime
View raw message