harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Varlamov" <alexey.v.varla...@gmail.com>
Subject Re: [build test] Integrate Emma into BTI 2.0 to get code coverage report
Date Thu, 27 Sep 2007 05:34:30 GMT
Sean,

There are a few more adaptors providing runtime, e.g. hdk,
make-snapshot, snapshot - patching all of them does not look feasible
IMO.

2007/9/27, Sean Qiu <sean.xx.qiu@gmail.com>:
> 2007/9/17, Stepan Mishura <stepan.mishura@gmail.com>:
> >
> > On 9/13/07, Alexey Varlamov wrote:
> > > Stepan,
> > >
> > > IMO current way of snapshot results publishing is a workaround
> > > basically, due to lack of adequate support in BTI to the moment. We
> > > should not hack snapshot adapter each time new suite/adapter is added
> > > or changed.
> > > So we need to extend BTI for more flexible and powerfull publishing
> > > facilities in some time (presumably after M3 competion), and then get
> > > rid of that custom publishers in snapshot.
> > >
> >
> > As I see the question is where this functionality (i.e. publishers)
> > should be placed.
> > I see only two variants - a test suite adaptor or snapshot (and emma)
> > adaptor. IMO the suite adaptor is not a proper place of such publisher
> > - it shouldn't be aware about snapshot testing or emma reports. All it
> > needs to know only how to run selected suite. Any other steps like
> > preparing a jre, a test suite's workspace or collecting reports
> > shouldn't be performed by the suite adaptor.
> >
> > I'd not say that by adding/removing adaptors for snapshot testing we
> > hack the snapshot adaptor. We'd rather hacked the suites adaptors by
> > bringing snapshots testing specific to them like:
> >     <condition property="is.snapshot">
> >         <equals arg1="${classlib-test.parameters.depends}"
> >                 arg2="snapshot"
> >                 trim="yes" />
> >     </condition>
> > ...
> >     <target name="-update-classlib-ws"
> >             if="is.snapshot"
> > <SNIP>
> >
> > I think this is IMO a hack indeed and we should find a way to move it
> > to the snapshots adaptor.
> >
> > > OTOH I agree, the way Sean suggests is too specific; I'd prefer keep
> > > in BTI most general functionalities.
> > > So there could be 2 adaptors, 1st instruments JRE and constitutes a
> > > "super-suite" for tests like Stepan suggested, and 2nd depends on all
> > > tests and generates summary report in the end. A couple of issues lies
> > > on this way, though:
> > > 1) How to configure emma for each suite, leaving an adaptor of that
> > > suite intact? I.e. which kernel-classes-list to take, where to store
> > > coverage report, etc - looks like Sean has some answers or ideas here;
> > > anyway there should not be more than light massaging of emma code.
> >
> > My impression was that we configure emma only once not for every
> > suite. Am I wrong?
> >
> > > Gathering total cumulative coverage report is enough, I think.
> > >
> > > 2) Currently BTI handles suites dependency strictly, always skipping
> > > dependent suite run if it's super-suite has failed. So if some test
> > > fails, coverage adapter won't be run. We may just add customization to
> > > BTI to enable simple run ordering vs strong dependency.
> > >
> >
> > I think 2 adaptors + changes in BTI are unnecessary and it will
> > complicate the testing logic. I'd suggest to keep all specific steps
> > (like updating suite to snapshot revision, instrumenting jre or
> > collecting reports) in snapshot or emma adaptors. And no splitting in
> > 2 adaptors. I think this can be done by providing custom CC
> > configuration file for suites.
> >
> > Ideas?
>
>
> We can add a flag parameter in drlvm's parameters.xml.
> This flag can disable the emma instrment by default.
> When we need a instrumented jre, we can just enable this flag. Any original
> logic of test running will not change at all.
> We only need a individual adaptor to collect the emma info after the test
> are finished within the  instrmented jre.
> There is no dependence introduced by this adaptor, since it will generate
> reports if any.
>
> By this mean, we only need one new adaptor to generate each test suite's
> coverage report.
> The instrument step is integrated into the drlvm adatpor as an optional.
> The cost is we have to download the emma during the installation of bti.
>
> Any commends?
>
> Thanks,
> > Stepan.
> >
> > > --
> > > Alexey
> > >
> > >
> > > 2007/9/13, Stepan Mishura <stepan.mishura@gmail.com>:
> > > > On 9/13/07, Sean Qiu wrote:
> > > > > We all know the scenario for emma is :
> > > > > 1) instument jre
> > > > > 2) run test within the instumented jre and generate coverage file
> > > > > 3) collect the coverage file and generate the coverage report
> > > > >
> > > > > For further study of emma, i have two concerns:
> > > > > 1. We have to maintain this task outside the adaptor, since it need
> > to run
> > > > > the (3) step after test.
> > > > >    This cannot be achieved elegantly by a single adaptor but add
a
> > new
> > > > > target in main script like "run-cc"
> > > > >     Agree? Commnets?
> > > > >
> > > > > 2. If we donot change the existing adaptor , the coverage file will
> > be
> > > > > placed in the dir where test is invoked.
> > > > >    I think we should locate them in a fixed place at least, like
> > > > > build/results/${test_suites}.
> > > > >    A alternative solution is put them all in a stand along dir, like
> > > > > build/emma
> > > > >    For both of them, we need add a new line in existing adaptor to
> > tell
> > > > > emma where we want to place our coverage file.
> > > > >    Is this modification acceptable?
> > > > >
> > > > > If nobody object, i will start to:
> > > > > 1) Add the emma target in main script
> > > > > 2) Try to put coverage file into a stand along dir
> > > > >
> > > > > Any suggestions are welcomed.
> > > > >
> > > >
> > > > I think that no changes to the framework or existing adaptors are
> > > > needed. Collection of coverage files and report generation for each
> > > > suite can by done by emma publisher (like snapshot testing does).
> > > >
> > > > So I'd implement emma adaptor in the way similar to the snapshot
> > > > testing, for example:
> > > >
> > > > 1) emma setup target - downloads and unpacks emma
> > > > 2) emma run target - instruments jre
> > > >
> > > > 3) emma adaptor depends on jre to be instrumented (it may be snapshot,
> > > > drlvm or hdk project)
> > > > (i.e. emma.parameters.depends=(snapshot|drlvm|hdk))
> > > >
> > > > 4) all suites depend on emma project.
> > > > ( <suite>.parameters.required.tested.runtime=${
> > emma.parameters.shared.instrumented.jre})
> > > >
> > > > 5) create custom config.xml for CC with emma publisher for suites
> > > > (<suite>..parameters.cc.config=${adaptors.dir}/emma/custom-config.xml)
> > > >
> > > > Thanks,
> > > > Stepan.
> > > >
> >
>
>
>
> --
> Sean Qiu
> China Software Development Lab, IBM
>

Mime
View raw message