harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Qiu" <sean.xx....@gmail.com>
Subject Re: [build test] Integrate Emma into BTI 2.0 to get code coverage report
Date Thu, 27 Sep 2007 06:43:04 GMT
I think patch one of them is enough.
Anyway, what we want is the coverage report for our latest classlib and
kernel classes, am i right ?

So your preference is two adaptors separately ?
Where should the adaptor get snapshot from ?  Make the snapshot source a
parameter in parameters.xml?

2007/9/27, Alexey Varlamov <alexey.v.varlamov@gmail.com>:
>
> Sean,
>
> There are a few more adaptors providing runtime, e.g. hdk,
> make-snapshot, snapshot - patching all of them does not look feasible
> IMO.
>
> 2007/9/27, Sean Qiu <sean.xx.qiu@gmail.com>:
> > 2007/9/17, Stepan Mishura <stepan.mishura@gmail.com>:
> > >
> > > On 9/13/07, Alexey Varlamov wrote:
> > > > Stepan,
> > > >
> > > > IMO current way of snapshot results publishing is a workaround
> > > > basically, due to lack of adequate support in BTI to the moment. We
> > > > should not hack snapshot adapter each time new suite/adapter is
> added
> > > > or changed.
> > > > So we need to extend BTI for more flexible and powerfull publishing
> > > > facilities in some time (presumably after M3 competion), and then
> get
> > > > rid of that custom publishers in snapshot.
> > > >
> > >
> > > As I see the question is where this functionality (i.e. publishers)
> > > should be placed.
> > > I see only two variants - a test suite adaptor or snapshot (and emma)
> > > adaptor. IMO the suite adaptor is not a proper place of such publisher
> > > - it shouldn't be aware about snapshot testing or emma reports. All it
> > > needs to know only how to run selected suite. Any other steps like
> > > preparing a jre, a test suite's workspace or collecting reports
> > > shouldn't be performed by the suite adaptor.
> > >
> > > I'd not say that by adding/removing adaptors for snapshot testing we
> > > hack the snapshot adaptor. We'd rather hacked the suites adaptors by
> > > bringing snapshots testing specific to them like:
> > >     <condition property="is.snapshot">
> > >         <equals arg1="${classlib-test.parameters.depends}"
> > >                 arg2="snapshot"
> > >                 trim="yes" />
> > >     </condition>
> > > ...
> > >     <target name="-update-classlib-ws"
> > >             if="is.snapshot"
> > > <SNIP>
> > >
> > > I think this is IMO a hack indeed and we should find a way to move it
> > > to the snapshots adaptor.
> > >
> > > > OTOH I agree, the way Sean suggests is too specific; I'd prefer keep
> > > > in BTI most general functionalities.
> > > > So there could be 2 adaptors, 1st instruments JRE and constitutes a
> > > > "super-suite" for tests like Stepan suggested, and 2nd depends on
> all
> > > > tests and generates summary report in the end. A couple of issues
> lies
> > > > on this way, though:
> > > > 1) How to configure emma for each suite, leaving an adaptor of that
> > > > suite intact? I.e. which kernel-classes-list to take, where to store
> > > > coverage report, etc - looks like Sean has some answers or ideas
> here;
> > > > anyway there should not be more than light massaging of emma code.
> > >
> > > My impression was that we configure emma only once not for every
> > > suite. Am I wrong?
> > >
> > > > Gathering total cumulative coverage report is enough, I think.
> > > >
> > > > 2) Currently BTI handles suites dependency strictly, always skipping
> > > > dependent suite run if it's super-suite has failed. So if some test
> > > > fails, coverage adapter won't be run. We may just add customization
> to
> > > > BTI to enable simple run ordering vs strong dependency.
> > > >
> > >
> > > I think 2 adaptors + changes in BTI are unnecessary and it will
> > > complicate the testing logic. I'd suggest to keep all specific steps
> > > (like updating suite to snapshot revision, instrumenting jre or
> > > collecting reports) in snapshot or emma adaptors. And no splitting in
> > > 2 adaptors. I think this can be done by providing custom CC
> > > configuration file for suites.
> > >
> > > Ideas?
> >
> >
> > We can add a flag parameter in drlvm's parameters.xml.
> > This flag can disable the emma instrment by default.
> > When we need a instrumented jre, we can just enable this flag. Any
> original
> > logic of test running will not change at all.
> > We only need a individual adaptor to collect the emma info after the
> test
> > are finished within the  instrmented jre.
> > There is no dependence introduced by this adaptor, since it will
> generate
> > reports if any.
> >
> > By this mean, we only need one new adaptor to generate each test suite's
> > coverage report.
> > The instrument step is integrated into the drlvm adatpor as an optional.
> > The cost is we have to download the emma during the installation of bti.
> >
> > Any commends?
> >
> > Thanks,
> > > Stepan.
> > >
> > > > --
> > > > Alexey
> > > >
> > > >
> > > > 2007/9/13, Stepan Mishura <stepan.mishura@gmail.com>:
> > > > > On 9/13/07, Sean Qiu wrote:
> > > > > > We all know the scenario for emma is :
> > > > > > 1) instument jre
> > > > > > 2) run test within the instumented jre and generate coverage
> file
> > > > > > 3) collect the coverage file and generate the coverage report
> > > > > >
> > > > > > For further study of emma, i have two concerns:
> > > > > > 1. We have to maintain this task outside the adaptor, since
it
> need
> > > to run
> > > > > > the (3) step after test.
> > > > > >    This cannot be achieved elegantly by a single adaptor but
add
> a
> > > new
> > > > > > target in main script like "run-cc"
> > > > > >     Agree? Commnets?
> > > > > >
> > > > > > 2. If we donot change the existing adaptor , the coverage file
> will
> > > be
> > > > > > placed in the dir where test is invoked.
> > > > > >    I think we should locate them in a fixed place at least,
like
> > > > > > build/results/${test_suites}.
> > > > > >    A alternative solution is put them all in a stand along dir,
> like
> > > > > > build/emma
> > > > > >    For both of them, we need add a new line in existing adaptor
> to
> > > tell
> > > > > > emma where we want to place our coverage file.
> > > > > >    Is this modification acceptable?
> > > > > >
> > > > > > If nobody object, i will start to:
> > > > > > 1) Add the emma target in main script
> > > > > > 2) Try to put coverage file into a stand along dir
> > > > > >
> > > > > > Any suggestions are welcomed.
> > > > > >
> > > > >
> > > > > I think that no changes to the framework or existing adaptors are
> > > > > needed. Collection of coverage files and report generation for
> each
> > > > > suite can by done by emma publisher (like snapshot testing does).
> > > > >
> > > > > So I'd implement emma adaptor in the way similar to the snapshot
> > > > > testing, for example:
> > > > >
> > > > > 1) emma setup target - downloads and unpacks emma
> > > > > 2) emma run target - instruments jre
> > > > >
> > > > > 3) emma adaptor depends on jre to be instrumented (it may be
> snapshot,
> > > > > drlvm or hdk project)
> > > > > (i.e. emma.parameters.depends=(snapshot|drlvm|hdk))
> > > > >
> > > > > 4) all suites depend on emma project.
> > > > > ( <suite>.parameters.required.tested.runtime=${
> > > emma.parameters.shared.instrumented.jre})
> > > > >
> > > > > 5) create custom config.xml for CC with emma publisher for suites
> > > > > (<suite>..parameters.cc.config=${adaptors.dir}/emma/custom-
> config.xml)
> > > > >
> > > > > Thanks,
> > > > > Stepan.
> > > > >
> > >
> >
> >
> >
> > --
> > Sean Qiu
> > China Software Development Lab, IBM
> >
>



-- 
Sean Qiu
China Software Development Lab, IBM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message