Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (athena.apache.org: domain of alexey.v.varlamov@gmail.com
 designates 209.85.198.191 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=ofuoqhS8d6SRNUJios+UhUGsl39sx8e8ls2oql6Zkin1Ktt1IY+Ub3CEhh8BnTEMOSSHq3jt4GuvwX8zK/puDc0GM45GuR6Q2GLsX6YntrGI/n6GQ68xGAhB3M4toLBF4B076XoRJv+sYq1QhWdqmf9Gs8Fp4yC0AYxDzxXeZnw=
Message-ID: <f944d83d0709130235i2d87dcb4y5b979f24f66c6aa4@mail.gmail.com>
Date: Thu, 13 Sep 2007 16:35:50 +0700
From: "Alexey Varlamov" <alexey.v.varlamov@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [build test] Integrate Emma into BTI 2.0 to get code coverage
 report
In-Reply-To: <6e47b64f0709122318t69886392oa81b3679dc179497@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <94d710af0709092338t72358427ua81d7e8d87ed1216@mail.gmail.com>
	 <6e47b64f0709102022j86cddb7te6be78a49ff5280a@mail.gmail.com>
	 <94d710af0709102253j6eeba934q6c0aa2b4c5212503@mail.gmail.com>
	 <6e47b64f0709112157o11d85245of4baecfca55273b@mail.gmail.com>
	 <94d710af0709122056t30b2632fxb5540c6863b67cc7@mail.gmail.com>
	 <6e47b64f0709122318t69886392oa81b3679dc179497@mail.gmail.com>

Stepan,

IMO current way of snapshot results publishing is a workaround
basically, due to lack of adequate support in BTI to the moment. We
should not hack snapshot adapter each time new suite/adapter is added
or changed.
So we need to extend BTI for more flexible and powerfull publishing
facilities in some time (presumably after M3 competion), and then get
rid of that custom publishers in snapshot.

OTOH I agree, the way Sean suggests is too specific; I'd prefer keep
in BTI most general functionalities.
So there could be 2 adaptors, 1st instruments JRE and constitutes a
"super-suite" for tests like Stepan suggested, and 2nd depends on all
tests and generates summary report in the end. A couple of issues lies
on this way, though:
1) How to configure emma for each suite, leaving an adaptor of that
suite intact? I.e. which kernel-classes-list to take, where to store
coverage report, etc - looks like Sean has some answers or ideas here;
anyway there should not be more than light massaging of emma code.
Gathering total cumulative coverage report is enough, I think.

2) Currently BTI handles suites dependency strictly, always skipping
dependent suite run if it's super-suite has failed. So if some test
fails, coverage adapter won't be run. We may just add customization to
BTI to enable simple run ordering vs strong dependency.

--
Alexey


2007/9/13, Stepan Mishura <stepan.mishura@gmail.com>:
> On 9/13/07, Sean Qiu wrote:
> > We all know the scenario for emma is :
> > 1) instument jre
> > 2) run test within the instumented jre and generate coverage file
> > 3) collect the coverage file and generate the coverage report
> >
> > For further study of emma, i have two concerns:
> > 1. We have to maintain this task outside the adaptor, since it need to run
> > the (3) step after test.
> >    This cannot be achieved elegantly by a single adaptor but add a new
> > target in main script like "run-cc"
> >     Agree? Commnets?
> >
> > 2. If we donot change the existing adaptor , the coverage file will be
> > placed in the dir where test is invoked.
> >    I think we should locate them in a fixed place at least, like
> > build/results/${test_suites}.
> >    A alternative solution is put them all in a stand along dir, like
> > build/emma
> >    For both of them, we need add a new line in existing adaptor to tell
> > emma where we want to place our coverage file.
> >    Is this modification acceptable?
> >
> > If nobody object, i will start to:
> > 1) Add the emma target in main script
> > 2) Try to put coverage file into a stand along dir
> >
> > Any suggestions are welcomed.
> >
>
> I think that no changes to the framework or existing adaptors are
> needed. Collection of coverage files and report generation for each
> suite can by done by emma publisher (like snapshot testing does).
>
> So I'd implement emma adaptor in the way similar to the snapshot
> testing, for example:
>
> 1) emma setup target - downloads and unpacks emma
> 2) emma run target - instruments jre
>
> 3) emma adaptor depends on jre to be instrumented (it may be snapshot,
> drlvm or hdk project)
> (i.e. emma.parameters.depends=(snapshot|drlvm|hdk))
>
> 4) all suites depend on emma project.
> ( <suite>.parameters.required.tested.runtime=${emma.parameters.shared.instrumented.jre})
>
> 5) create custom config.xml for CC with emma publisher for suites
> (<suite>..parameters.cc.config=${adaptors.dir}/emma/custom-config.xml)
>
> Thanks,
> Stepan.
>