Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of Alex.Huang@citrix.com
 designates 66.165.176.89 as permitted sender)
From: Alex Huang <Alex.Huang@citrix.com>
To: "cloudstack-dev@incubator.apache.org"
	<cloudstack-dev@incubator.apache.org>
Date: Mon, 7 Jan 2013 16:00:11 -0800
Subject: RE: [Discuss] Integration test 
Thread-Topic: [Discuss] Integration test 
Thread-Index: Ac3tI6eaDLl2wJRfTOu/b0tW59IetwAB5DMQAAAwi4AAAHcngA==
Message-ID: 
 <B1DF26ECC0458748AC97CECE2DA98D41012FA385ECC2@SJCPMAILBOX01.citrite.net>
References: 
 <C66C814C1ABFA8449FF65CB44953A8B4012CBF3A573E@SJCPMAILBOX01.citrite.net>
 <7914B38A4445B34AA16EB9F1352942F1012F1461F817@SJCPMAILBOX01.citrite.net>
 <C66C814C1ABFA8449FF65CB44953A8B4012CBF3A573F@SJCPMAILBOX01.citrite.net>
In-Reply-To: 
 <C66C814C1ABFA8449FF65CB44953A8B4012CBF3A573F@SJCPMAILBOX01.citrite.net>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Marvin is testing for an integrated system.

Here, we're talking about testing an integration contract.  This means prov=
iding tests that represents how a plugin would be used by cloudstack.  A pl=
ugin writer should be able to run their plugin within this test suite WITHO=
UT cloudstack and get some guarantee that their plugin supports the interfa=
ce correctly and won't break anything.  This is purely developer level inte=
gration test.=20

This means that if someone changes the logic in the subsystem, they are als=
o responsible for changing the integration test cases.

Note that this type of testing cannot replace marvin.

--Alex

> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: Monday, January 07, 2013 3:13 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [Discuss] Integration test
>=20
> No, marvin is suitable for API test, can't test java maven project direct=
ly. Here,
> the integration test, is more about test case for a set of java maven pro=
jects,
> so it must be tested by using java test framework, either junit or testNG=
. For
> unit test, test for an individual maven project, it's ok to use Junit. Bu=
t for
> integration test, I'd prefer testNG.
>=20
> > -----Original Message-----
> > From: Sudha Ponnaganti [mailto:sudha.ponnaganti@citrix.com]
> > Sent: Monday, January 07, 2013 3:07 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: [Discuss] Integration test
> >
> > Edison,
> >
> > Current python framework ( marvin) covers several of the
> components/sub-
> > modules and system ( end to end) tests. Functionally it is serving the
> purpose.
> > Is this proposal to substitute that as well as current Junit framework?=
?
> >
> > Thanks
> > /Sudha
> >
> >
> > -----Original Message-----
> > From: Edison Su [mailto:Edison.su@citrix.com]
> > Sent: Monday, January 07, 2013 2:11 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: [Discuss] Integration test
> >
> > Hi All,
> >      As Cloudstack is becoming more modularized, we can start thinking
> about
> > how to test these components.
> >      The organization of maven projects in cloudstack. At lowest level,=
 it's a
> > maven project. The modularized component, means a group of maven
> > projects. The subsystem, means a logical group of  components. A system=
,
> > will be build on a lot of subsystems. For example, there will be a syst=
em
> > called, cloud-engine, which is a server, provides restful API service t=
o other
> > systems(e.g API server). Cloud-engine will have a set of subsystems:
> > computing, storage, network etc. For each subsystem, will have a group =
of
> > components: e.g. storage subsystem will include volume/image/snapshot .=
.
> > components. For each component, will have a set of maven projects,  for
> > example, volume component will have a maven project, called storage-
> > volume, and a lot of plugin projects, such as (storage-volume-plugin-
> solidfire,
> > storage-volume-plugin-netapp etc).
> >      Based on above hierarchy, the integration test will happen at diff=
erent
> > levels: at component level, at subsystem level, at system level etc. I =
want
> to
> > bring up the discuss about how to organize integration tests, and which
> > integration test framework should we use.
> >     First, how to organize integration tests? The organization will be
> > affected/constrained by the dependency between maven projects you
> want
> > to test.
> >           1. Will the integration test at each level have its own maven=
 project?
> Or
> > we put all the integration test cases into one giant maven project? I'd
> prefer
> > one maven project at one level, as it looks like more cleaner. For exam=
ple,
> > there will be a storage-integration-test maven project, which depends o=
n
> > storage-volume-integration-test, storage-image-integration-test, and so
> on,
> > while storage-volume-integration-test will depend on storage-volume,
> > storage-volume-plugin-solidfire, and storage-volume-plugin-netapp and s=
o
> > on. When you write test case at each level, you only focus on the limit=
ed
> area
> > you want to test: the design of test case and the configuration file of=
 each
> > test case, will/can only be specific to the area. Hope it will make wri=
ting test
> > case easier.
> >           2. Which test framework we should use? Junit or testNG? I was=
 sold to
> > testNG by its feature sets: parametrized test case, group test cases,
> running
> > test case in parallel at different level(method, class etc), test case
> > dependency etc. You can find out more information about testNG at
> > http://kaczanowscy.pl/tomek/sites/default/files/testng_vs_junit.txt.sli=
dy_.
> > html#(1).
> > 	But in order to integrate testNG with Spring, need a little bit coding=
.
> > For example, may need to add a listener in testNG, to inject @DB
> annotation,
> > and need to make sure mockito will work. I created a base testNG class:
> > CloudStackTestNGBase, which can read configuration parameters from a
> > testNG config xml file, setup @DB context for each test case etc.
> >
> >    Comments/feedback are welcome!
> >
> >