Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (athena.apache.org: domain of alexey.v.varlamov@gmail.com
 designates 209.85.198.184 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=KUxB9pMg5/2FXVJ1O6egJu0E2mTHBKAb1yfxcJnOirKCV0iV+Uc6nu4BvwZJOi3MmlTVhxOIwp0786LSHNmUdRcBqMprMh+VsPYG65juNBS04MovpHs/hFeDi3lLXCpt0Tx0VLW4KQpNU/kjwsCRqfgOEAO8zCigGNHZVwPg3T4=
Message-ID: <f944d83d0709270318m6c804634md44c62501155e4f5@mail.gmail.com>
Date: Thu, 27 Sep 2007 17:18:30 +0700
From: "Alexey Varlamov" <alexey.v.varlamov@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [build test] Integrate Emma into BTI 2.0 to get code coverage
 report
In-Reply-To: <94d710af0709262343j26a8256euccb1fd4693311cc@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <94d710af0709092338t72358427ua81d7e8d87ed1216@mail.gmail.com>
	 <94d710af0709102253j6eeba934q6c0aa2b4c5212503@mail.gmail.com>
	 <6e47b64f0709112157o11d85245of4baecfca55273b@mail.gmail.com>
	 <94d710af0709122056t30b2632fxb5540c6863b67cc7@mail.gmail.com>
	 <6e47b64f0709122318t69886392oa81b3679dc179497@mail.gmail.com>
	 <f944d83d0709130235i2d87dcb4y5b979f24f66c6aa4@mail.gmail.com>
	 <6e47b64f0709170644lda154eev81cb7004c78244c6@mail.gmail.com>
	 <94d710af0709262213s117f59a0v76d569835dc5cb60@mail.gmail.com>
	 <f944d83d0709262234j606a9980uaedcc2ff18b522e@mail.gmail.com>
	 <94d710af0709262343j26a8256euccb1fd4693311cc@mail.gmail.com>

2007/9/27, Sean Qiu <sean.xx.qiu@gmail.com>:
> I think patch one of them is enough.
> Anyway, what we want is the coverage report for our latest classlib and
> kernel classes, am i right ?

II'd prefer wider range of targets - IMO it's much more interesting to
obtain coverage by application scenarios. Then we could better balance
integrity, pre-commit, snapshot testing, etc.

>
> So your preference is two adaptors separately ?
> Where should the adaptor get snapshot from ?  Make the snapshot source a
> parameter in parameters.xml?
>
> 2007/9/27, Alexey Varlamov <alexey.v.varlamov@gmail.com>:
> >
> > Sean,
> >
> > There are a few more adaptors providing runtime, e.g. hdk,
> > make-snapshot, snapshot - patching all of them does not look feasible
> > IMO.
> >
> > 2007/9/27, Sean Qiu <sean.xx.qiu@gmail.com>:
> > > 2007/9/17, Stepan Mishura <stepan.mishura@gmail.com>:
> > > >
> > > > On 9/13/07, Alexey Varlamov wrote:
> > > > > Stepan,
> > > > >
> > > > > IMO current way of snapshot results publishing is a workaround
> > > > > basically, due to lack of adequate support in BTI to the moment. We
> > > > > should not hack snapshot adapter each time new suite/adapter is
> > added
> > > > > or changed.
> > > > > So we need to extend BTI for more flexible and powerfull publishing
> > > > > facilities in some time (presumably after M3 competion), and then
> > get
> > > > > rid of that custom publishers in snapshot.
> > > > >
> > > >
> > > > As I see the question is where this functionality (i.e. publishers)
> > > > should be placed.
> > > > I see only two variants - a test suite adaptor or snapshot (and emma)
> > > > adaptor. IMO the suite adaptor is not a proper place of such publisher
> > > > - it shouldn't be aware about snapshot testing or emma reports. All it
> > > > needs to know only how to run selected suite. Any other steps like
> > > > preparing a jre, a test suite's workspace or collecting reports
> > > > shouldn't be performed by the suite adaptor.
> > > >
> > > > I'd not say that by adding/removing adaptors for snapshot testing we
> > > > hack the snapshot adaptor. We'd rather hacked the suites adaptors by
> > > > bringing snapshots testing specific to them like:
> > > >     <condition property="is.snapshot">
> > > >         <equals arg1="${classlib-test.parameters.depends}"
> > > >                 arg2="snapshot"
> > > >                 trim="yes" />
> > > >     </condition>
> > > > ...
> > > >     <target name="-update-classlib-ws"
> > > >             if="is.snapshot"
> > > > <SNIP>
> > > >
> > > > I think this is IMO a hack indeed and we should find a way to move it
> > > > to the snapshots adaptor.
> > > >
> > > > > OTOH I agree, the way Sean suggests is too specific; I'd prefer keep
> > > > > in BTI most general functionalities.
> > > > > So there could be 2 adaptors, 1st instruments JRE and constitutes a
> > > > > "super-suite" for tests like Stepan suggested, and 2nd depends on
> > all
> > > > > tests and generates summary report in the end. A couple of issues
> > lies
> > > > > on this way, though:
> > > > > 1) How to configure emma for each suite, leaving an adaptor of that
> > > > > suite intact? I.e. which kernel-classes-list to take, where to store
> > > > > coverage report, etc - looks like Sean has some answers or ideas
> > here;
> > > > > anyway there should not be more than light massaging of emma code.
> > > >
> > > > My impression was that we configure emma only once not for every
> > > > suite. Am I wrong?
> > > >
> > > > > Gathering total cumulative coverage report is enough, I think.
> > > > >
> > > > > 2) Currently BTI handles suites dependency strictly, always skipping
> > > > > dependent suite run if it's super-suite has failed. So if some test
> > > > > fails, coverage adapter won't be run. We may just add customization
> > to
> > > > > BTI to enable simple run ordering vs strong dependency.
> > > > >
> > > >
> > > > I think 2 adaptors + changes in BTI are unnecessary and it will
> > > > complicate the testing logic. I'd suggest to keep all specific steps
> > > > (like updating suite to snapshot revision, instrumenting jre or
> > > > collecting reports) in snapshot or emma adaptors. And no splitting in
> > > > 2 adaptors. I think this can be done by providing custom CC
> > > > configuration file for suites.
> > > >
> > > > Ideas?
> > >
> > >
> > > We can add a flag parameter in drlvm's parameters.xml.
> > > This flag can disable the emma instrment by default.
> > > When we need a instrumented jre, we can just enable this flag. Any
> > original
> > > logic of test running will not change at all.
> > > We only need a individual adaptor to collect the emma info after the
> > test
> > > are finished within the  instrmented jre.
> > > There is no dependence introduced by this adaptor, since it will
> > generate
> > > reports if any.
> > >
> > > By this mean, we only need one new adaptor to generate each test suite's
> > > coverage report.
> > > The instrument step is integrated into the drlvm adatpor as an optional.
> > > The cost is we have to download the emma during the installation of bti.
> > >
> > > Any commends?
> > >
> > > Thanks,
> > > > Stepan.
> > > >
> > > > > --
> > > > > Alexey
> > > > >
> > > > >
> > > > > 2007/9/13, Stepan Mishura <stepan.mishura@gmail.com>:
> > > > > > On 9/13/07, Sean Qiu wrote:
> > > > > > > We all know the scenario for emma is :
> > > > > > > 1) instument jre
> > > > > > > 2) run test within the instumented jre and generate coverage
> > file
> > > > > > > 3) collect the coverage file and generate the coverage report
> > > > > > >
> > > > > > > For further study of emma, i have two concerns:
> > > > > > > 1. We have to maintain this task outside the adaptor, since it
> > need
> > > > to run
> > > > > > > the (3) step after test.
> > > > > > >    This cannot be achieved elegantly by a single adaptor but add
> > a
> > > > new
> > > > > > > target in main script like "run-cc"
> > > > > > >     Agree? Commnets?
> > > > > > >
> > > > > > > 2. If we donot change the existing adaptor , the coverage file
> > will
> > > > be
> > > > > > > placed in the dir where test is invoked.
> > > > > > >    I think we should locate them in a fixed place at least, like
> > > > > > > build/results/${test_suites}.
> > > > > > >    A alternative solution is put them all in a stand along dir,
> > like
> > > > > > > build/emma
> > > > > > >    For both of them, we need add a new line in existing adaptor
> > to
> > > > tell
> > > > > > > emma where we want to place our coverage file.
> > > > > > >    Is this modification acceptable?
> > > > > > >
> > > > > > > If nobody object, i will start to:
> > > > > > > 1) Add the emma target in main script
> > > > > > > 2) Try to put coverage file into a stand along dir
> > > > > > >
> > > > > > > Any suggestions are welcomed.
> > > > > > >
> > > > > >
> > > > > > I think that no changes to the framework or existing adaptors are
> > > > > > needed. Collection of coverage files and report generation for
> > each
> > > > > > suite can by done by emma publisher (like snapshot testing does).
> > > > > >
> > > > > > So I'd implement emma adaptor in the way similar to the snapshot
> > > > > > testing, for example:
> > > > > >
> > > > > > 1) emma setup target - downloads and unpacks emma
> > > > > > 2) emma run target - instruments jre
> > > > > >
> > > > > > 3) emma adaptor depends on jre to be instrumented (it may be
> > snapshot,
> > > > > > drlvm or hdk project)
> > > > > > (i.e. emma.parameters.depends=(snapshot|drlvm|hdk))
> > > > > >
> > > > > > 4) all suites depend on emma project.
> > > > > > ( <suite>.parameters.required.tested.runtime=${
> > > > emma.parameters.shared.instrumented.jre})
> > > > > >
> > > > > > 5) create custom config.xml for CC with emma publisher for suites
> > > > > > (<suite>..parameters.cc.config=${adaptors.dir}/emma/custom-
> > config.xml)
> > > > > >
> > > > > > Thanks,
> > > > > > Stepan.
> > > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Sean Qiu
> > > China Software Development Lab, IBM
> > >
> >
>
>
>
> --
> Sean Qiu
> China Software Development Lab, IBM
>