harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Qiu" <sean.xx....@gmail.com>
Subject Re: [build test] Integrate Emma into BTI 2.0 to get code coverage report
Date Thu, 27 Sep 2007 05:13:12 GMT
2007/9/17, Stepan Mishura <stepan.mishura@gmail.com>:
>
> On 9/13/07, Alexey Varlamov wrote:
> > Stepan,
> >
> > IMO current way of snapshot results publishing is a workaround
> > basically, due to lack of adequate support in BTI to the moment. We
> > should not hack snapshot adapter each time new suite/adapter is added
> > or changed.
> > So we need to extend BTI for more flexible and powerfull publishing
> > facilities in some time (presumably after M3 competion), and then get
> > rid of that custom publishers in snapshot.
> >
>
> As I see the question is where this functionality (i.e. publishers)
> should be placed.
> I see only two variants - a test suite adaptor or snapshot (and emma)
> adaptor. IMO the suite adaptor is not a proper place of such publisher
> - it shouldn't be aware about snapshot testing or emma reports. All it
> needs to know only how to run selected suite. Any other steps like
> preparing a jre, a test suite's workspace or collecting reports
> shouldn't be performed by the suite adaptor.
>
> I'd not say that by adding/removing adaptors for snapshot testing we
> hack the snapshot adaptor. We'd rather hacked the suites adaptors by
> bringing snapshots testing specific to them like:
>     <condition property="is.snapshot">
>         <equals arg1="${classlib-test.parameters.depends}"
>                 arg2="snapshot"
>                 trim="yes" />
>     </condition>
> ...
>     <target name="-update-classlib-ws"
>             if="is.snapshot"
> <SNIP>
>
> I think this is IMO a hack indeed and we should find a way to move it
> to the snapshots adaptor.
>
> > OTOH I agree, the way Sean suggests is too specific; I'd prefer keep
> > in BTI most general functionalities.
> > So there could be 2 adaptors, 1st instruments JRE and constitutes a
> > "super-suite" for tests like Stepan suggested, and 2nd depends on all
> > tests and generates summary report in the end. A couple of issues lies
> > on this way, though:
> > 1) How to configure emma for each suite, leaving an adaptor of that
> > suite intact? I.e. which kernel-classes-list to take, where to store
> > coverage report, etc - looks like Sean has some answers or ideas here;
> > anyway there should not be more than light massaging of emma code.
>
> My impression was that we configure emma only once not for every
> suite. Am I wrong?
>
> > Gathering total cumulative coverage report is enough, I think.
> >
> > 2) Currently BTI handles suites dependency strictly, always skipping
> > dependent suite run if it's super-suite has failed. So if some test
> > fails, coverage adapter won't be run. We may just add customization to
> > BTI to enable simple run ordering vs strong dependency.
> >
>
> I think 2 adaptors + changes in BTI are unnecessary and it will
> complicate the testing logic. I'd suggest to keep all specific steps
> (like updating suite to snapshot revision, instrumenting jre or
> collecting reports) in snapshot or emma adaptors. And no splitting in
> 2 adaptors. I think this can be done by providing custom CC
> configuration file for suites.
>
> Ideas?


We can add a flag parameter in drlvm's parameters.xml.
This flag can disable the emma instrment by default.
When we need a instrumented jre, we can just enable this flag. Any original
logic of test running will not change at all.
We only need a individual adaptor to collect the emma info after the test
are finished within the  instrmented jre.
There is no dependence introduced by this adaptor, since it will generate
reports if any.

By this mean, we only need one new adaptor to generate each test suite's
coverage report.
The instrument step is integrated into the drlvm adatpor as an optional.
The cost is we have to download the emma during the installation of bti.

Any commends?

Thanks,
> Stepan.
>
> > --
> > Alexey
> >
> >
> > 2007/9/13, Stepan Mishura <stepan.mishura@gmail.com>:
> > > On 9/13/07, Sean Qiu wrote:
> > > > We all know the scenario for emma is :
> > > > 1) instument jre
> > > > 2) run test within the instumented jre and generate coverage file
> > > > 3) collect the coverage file and generate the coverage report
> > > >
> > > > For further study of emma, i have two concerns:
> > > > 1. We have to maintain this task outside the adaptor, since it need
> to run
> > > > the (3) step after test.
> > > >    This cannot be achieved elegantly by a single adaptor but add a
> new
> > > > target in main script like "run-cc"
> > > >     Agree? Commnets?
> > > >
> > > > 2. If we donot change the existing adaptor , the coverage file will
> be
> > > > placed in the dir where test is invoked.
> > > >    I think we should locate them in a fixed place at least, like
> > > > build/results/${test_suites}.
> > > >    A alternative solution is put them all in a stand along dir, like
> > > > build/emma
> > > >    For both of them, we need add a new line in existing adaptor to
> tell
> > > > emma where we want to place our coverage file.
> > > >    Is this modification acceptable?
> > > >
> > > > If nobody object, i will start to:
> > > > 1) Add the emma target in main script
> > > > 2) Try to put coverage file into a stand along dir
> > > >
> > > > Any suggestions are welcomed.
> > > >
> > >
> > > I think that no changes to the framework or existing adaptors are
> > > needed. Collection of coverage files and report generation for each
> > > suite can by done by emma publisher (like snapshot testing does).
> > >
> > > So I'd implement emma adaptor in the way similar to the snapshot
> > > testing, for example:
> > >
> > > 1) emma setup target - downloads and unpacks emma
> > > 2) emma run target - instruments jre
> > >
> > > 3) emma adaptor depends on jre to be instrumented (it may be snapshot,
> > > drlvm or hdk project)
> > > (i.e. emma.parameters.depends=(snapshot|drlvm|hdk))
> > >
> > > 4) all suites depend on emma project.
> > > ( <suite>.parameters.required.tested.runtime=${
> emma.parameters.shared.instrumented.jre})
> > >
> > > 5) create custom config.xml for CC with emma publisher for suites
> > > (<suite>..parameters.cc.config=${adaptors.dir}/emma/custom-config.xml)
> > >
> > > Thanks,
> > > Stepan.
> > >
>



-- 
Sean Qiu
China Software Development Lab, IBM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message