harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Qiu" <sean.xx....@gmail.com>
Subject Re: [build test] Integrate Emma into BTI 2.0 to get code coverage report
Date Wed, 26 Sep 2007 06:00:30 GMT
2007/9/17, Stepan Mishura <stepan.mishura@gmail.com>:
>
> On 9/13/07, Alexey Varlamov wrote:
> > Stepan,
> >
> > IMO current way of snapshot results publishing is a workaround
> > basically, due to lack of adequate support in BTI to the moment. We
> > should not hack snapshot adapter each time new suite/adapter is added
> > or changed.
> > So we need to extend BTI for more flexible and powerfull publishing
> > facilities in some time (presumably after M3 competion), and then get
> > rid of that custom publishers in snapshot.
> >
>
> As I see the question is where this functionality (i.e. publishers)
> should be placed.
> I see only two variants - a test suite adaptor or snapshot (and emma)
> adaptor. IMO the suite adaptor is not a proper place of such publisher
> - it shouldn't be aware about snapshot testing or emma reports. All it
> needs to know only how to run selected suite. Any other steps like
> preparing a jre, a test suite's workspace or collecting reports
> shouldn't be performed by the suite adaptor.
>
> I'd not say that by adding/removing adaptors for snapshot testing we
> hack the snapshot adaptor. We'd rather hacked the suites adaptors by
> bringing snapshots testing specific to them like:
>     <condition property="is.snapshot">
>         <equals arg1="${classlib-test.parameters.depends}"
>                 arg2="snapshot"
>                 trim="yes" />
>     </condition>
> ...
>     <target name="-update-classlib-ws"
>             if="is.snapshot"
> <SNIP>
>
> I think this is IMO a hack indeed and we should find a way to move it
> to the snapshots adaptor.
>
> > OTOH I agree, the way Sean suggests is too specific; I'd prefer keep
> > in BTI most general functionalities.
> > So there could be 2 adaptors, 1st instruments JRE and constitutes a
> > "super-suite" for tests like Stepan suggested, and 2nd depends on all
> > tests and generates summary report in the end. A couple of issues lies
> > on this way, though:
> > 1) How to configure emma for each suite, leaving an adaptor of that
> > suite intact? I.e. which kernel-classes-list to take, where to store
> > coverage report, etc - looks like Sean has some answers or ideas here;
> > anyway there should not be more than light massaging of emma code.
>
> My impression was that we configure emma only once not for every
> suite. Am I wrong?
>
> > Gathering total cumulative coverage report is enough, I think.
> >
> > 2) Currently BTI handles suites dependency strictly, always skipping
> > dependent suite run if it's super-suite has failed. So if some test
> > fails, coverage adapter won't be run. We may just add customization to
> > BTI to enable simple run ordering vs strong dependency.
> >
>
> I think 2 adaptors + changes in BTI are unnecessary and it will
> complicate the testing logic. I'd suggest to keep all specific steps
> (like updating suite to snapshot revision, instrumenting jre or
> collecting reports) in snapshot or emma adaptors. And no splitting in
> 2 adaptors. I think this can be done by providing custom CC
> configuration file for suites.
>
> Ideas?


I'm not sure I get your point except your objection to 2 adaptors for emma.
Could you please explain it in details ?

Thanks,
> Stepan.
>
> > --
> > Alexey
> >
> >
> > 2007/9/13, Stepan Mishura <stepan.mishura@gmail.com>:
> > > On 9/13/07, Sean Qiu wrote:
> > > > We all know the scenario for emma is :
> > > > 1) instument jre
> > > > 2) run test within the instumented jre and generate coverage file
> > > > 3) collect the coverage file and generate the coverage report
> > > >
> > > > For further study of emma, i have two concerns:
> > > > 1. We have to maintain this task outside the adaptor, since it need
> to run
> > > > the (3) step after test.
> > > >    This cannot be achieved elegantly by a single adaptor but add a
> new
> > > > target in main script like "run-cc"
> > > >     Agree? Commnets?
> > > >
> > > > 2. If we donot change the existing adaptor , the coverage file will
> be
> > > > placed in the dir where test is invoked.
> > > >    I think we should locate them in a fixed place at least, like
> > > > build/results/${test_suites}.
> > > >    A alternative solution is put them all in a stand along dir, like
> > > > build/emma
> > > >    For both of them, we need add a new line in existing adaptor to
> tell
> > > > emma where we want to place our coverage file.
> > > >    Is this modification acceptable?
> > > >
> > > > If nobody object, i will start to:
> > > > 1) Add the emma target in main script
> > > > 2) Try to put coverage file into a stand along dir
> > > >
> > > > Any suggestions are welcomed.
> > > >
> > >
> > > I think that no changes to the framework or existing adaptors are
> > > needed. Collection of coverage files and report generation for each
> > > suite can by done by emma publisher (like snapshot testing does).
> > >
> > > So I'd implement emma adaptor in the way similar to the snapshot
> > > testing, for example:
> > >
> > > 1) emma setup target - downloads and unpacks emma
> > > 2) emma run target - instruments jre
> > >
> > > 3) emma adaptor depends on jre to be instrumented (it may be snapshot,
> > > drlvm or hdk project)
> > > (i.e. emma.parameters.depends=(snapshot|drlvm|hdk))
> > >
> > > 4) all suites depend on emma project.
> > > ( <suite>.parameters.required.tested.runtime=${
> emma.parameters.shared.instrumented.jre})
> > >
> > > 5) create custom config.xml for CC with emma publisher for suites
> > > (<suite>..parameters.cc.config=${adaptors.dir}/emma/custom-config.xml)
> > >
> > > Thanks,
> > > Stepan.
> > >
>



-- 
Sean Qiu
China Software Development Lab, IBM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message