Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: error (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <20130724065409.GA7695@cloud-2.local>
References: <20130724055647.GA5851@cloud-2.local>
	<CA+2rt43p2StzJ7fVsNhxzOp7EKC-Hy1AP2Rt2Zd0oxVe2zX1NQ@mail.gmail.com>
	<20130724065409.GA7695@cloud-2.local>
Date: Wed, 24 Jul 2013 13:45:13 -0700
Message-ID: 
 <CA+2rt41Jhw2LN4QU_oigq+U2qbkuuVPB9tyw5gYTne19wKyicg@mail.gmail.com>
Subject: Re: Automated Tests: Account Cleanup in test modules
From: Sheng Yang <sheng@yasker.org>
To: "<dev@cloudstack.apache.org>" <dev@cloudstack.apache.org>
Content-Type: multipart/alternative; boundary=047d7bacbce87e931704e247fc08

--047d7bacbce87e931704e247fc08
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jul 23, 2013 at 11:54 PM, Prasanna Santhanam <tsp@apache.org> wrote:

> On Tue, Jul 23, 2013 at 11:20:22PM -0700, Sheng Yang wrote:
> > About the patch I committed, in fact it's not moving. It's fixing.
> Because:
> > 1. The logical reason is, currently in many cases(which I committed the
> > patches) the account is already create per test case rather than per
> > module. And if account is created in setUp() rather than setUpClass(),
> the
> > clean up would be in tearDown() rather than tearDownClass(), because
> > they're useless anyway after setUp create another account and more VMs.
> My
> > patch didn't move account creation from setUpClass().
>
> Sheng - thanks I saw your patches more clearly now. The diffs mislead
> me that they were perhaps moved. You're right that the cleanup
> list should add the account in the locality of where it was created ie
> per-test or per-module as the case may be.
>
> > 2. The more direct reason is too many test cases in regression test
> failed
> > just because lack of resource to create new deployment. E.g.
> > https://issues.apache.org/jira/browse/CLOUDSTACK-3643 . Whenever I saw
> > "Fail to create VPC" or "Fail to deploy VM", mostly it because the
> testing
> > setup is overloaded.
>
> Yes this is a concern. We are mislead by resource exhaustion to be a
> failure in the test. Do you think the tests could be better organized
> when run in a group against a limited resource deployment?
>

I can think of following things:
1. The maximum parallel execution test cases number should leave some(but
not too much) margin for wrong test cases.
2. Every test case should take care of it's own resources and release them
ASAP after they're no longer in use.
3. Reduce the time out for failure operation. E.g. we don't need to wait
tens of minutes for SSH timeout.
4. Check for unnecessary and duplicate test cases.

If we can get test cases running faster, then they can release the
resources sooner.


> >
> > I know VM creating is very time consuming, and reuse the account and VM
> is
> > really nice! But the fact is many of current test cases is not written in
> > this way. So I think we should release the resource as soon as it's
> > obsolete.
> >
> Yup that's the way it was intended. We probably deviated from there
> and will fix these tests during this sprint.
>
> > About reusing the already deployed VM and account, it would be a
> > case-by-case issue. For example, TestAddVmToSubDomain case in
> test_accounts
> > do create the two VMs shared across the module(class), then tear it down
> > when in tearDownClass(). But sometime there is not a easy way to do so. I
> > think that's why there are not many VirtualMachine.create() happened in
> > setUpClass().
> >
>
> Agreed on this. Much of this information is within the test cases and
> until you drill down in the scenario you won't be aware of it. But
> in general the guidelines probably were not clear (I got questions
> from others working on the tests) so it was better to spell them out.
>
> It's time to write up these guidelines in our wiki which has some
> basic guidelines already:
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python#Marvin-TestingwithPython-Guidelinestochoosescenariosforintegration


Yes, I've add some clean up guideline to it.

--Sheng

>
>
> Thanks for sharing your thoughts,
>
>
> > --Sheng
> >
> >
> > On Tue, Jul 23, 2013 at 10:56 PM, Prasanna Santhanam <tsp@apache.org>
> wrote:
> >
> > > In the test modules when you debug you will notice that accounts are
> > > created once per module in setUpClass() all resources created within
> > > it and tearDownClass() destroys the account initiating cleanup. All
> > > the resources are appended to a cleanup [] list and deleted in
> > > appropriate order at the end of the test module in tearDownClass()
> > >
> > > There are a few reasons for doing this for all the tests in the class
> > > at once as opposed to doing it for every test.
> > >
> > > Modules are grouped by feature eg: test_tags does tags related
> > > tests. And more often than not all tests in the module share similar
> > > patterns in the set of API calls made to achieve a test scenario.
> > >
> > > If an account was created per test the overhead of cleaning up is much
> > > higher than when cleaning up once per module. Because every new
> > > account needs a new VR for the first VM deployed in it. And almost
> > > every test will deploy a VM. So this slows down test run significantly
> > > and eats up resources like VLANs very quickly which are needed for
> > > every account.
> > >
> > > I saw a few fixes that moved the resource cleanup from tearDownClass()
> > > to tearDown and that prompted me to send this email. Hope this makes
> > > sense. I'd like to hear other's thoughts on how best to accomplish all
> > > the tests in the most optimum way without hurting resources on a
> > > deployment.
> > >
> > > Also - on the test infrastructure on jenkins.buildacloud.org there is
> > > no way to timeout a specific test if it takes up more time than
> > > necessary. So I'm going to introduce a timeout plugin in nose that
> > > will abort the test if it takes longer than 1/2hr. I think this should
> > > help weeding out test that do arbitrary 'sleeps' or wait for very long
> > > cleanup operations. That way we should be able to optimize the test as
> > > well
> > >
> > > Thoughts?
> > >
> > > --
> > > Prasanna.,
> > >
> > > ------------------------
> > > Powered by BigRock.com
> > >
> > >
>
> --
> Prasanna.,
>
> ------------------------
> Powered by BigRock.com
>
>

--047d7bacbce87e931704e247fc08--