cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sudha Ponnaganti <sudha.ponnaga...@citrix.com>
Subject RE: Tiered Quality
Date Sat, 02 Nov 2013 13:35:17 GMT
That is only for Unit tests - we need to instrument code coverage for BVTs and Regressions
i.e integration tests.  We are pursuing this in our lab. If we get any break through will
post it to the forum. Because of customized nature of automation framework there are few challenges
there.

________________________________________
From: Laszlo Hornyak [laszlo.hornyak@gmail.com]
Sent: Friday, November 01, 2013 10:48 AM
To: dev@cloudstack.apache.org
Subject: Re: Tiered Quality

I have heard about commercial tools that do more advanced coverage
tracking. But if you think in open source, not sure Sonar really has an
alternative. It is pretty cool anyway.
Btw the overall code coverage is about 3.6%, probaly it is not worth trying
something more advanced for that much.


On Thu, Oct 31, 2013 at 9:12 PM, Daan Hoogland <daan.hoogland@gmail.com>wrote:

> one note on testing guys,
>
> I see that the analysis site give lines-coverage and branch-coverage.
> I don't see anything on distinct paths. What I mean is that the
> program
> if(a)
>  A
> else
>  B
> if(b)
>  C
> else
>  D
> if(c)
>  E
> else
>  F
> has eight (2^3) distict paths. It is not enough to show that
> A,B,C,D,E,F are all hit and hance every line and branch. Also all
> combinations of a/!a and b/!b and c/!c need to be hit.
>
> Now I am not saying that we should not score our code if not in this
> way but it is kind of kidding ourselves if we don't face up to the
> fact that coverage of lines of code or branches is not a completeness
> criterium of some kind. I don't know if any of the mentioned tools
> does analysis this thorough. But if any does we should go for that
> one.
>
> €0,02
> Daan
>
> On Tue, Oct 29, 2013 at 2:21 AM, Darren Shepherd
> <darren.s.shepherd@gmail.com> wrote:
> > Starting with the honor system might be good.  It's not so easy some
> times to relate lines of code to functionality.  Also just because it hits
> a line of code doesn't mean it's really tested.
> >
> > Can't we just get people to just put a check mark on some table in the
> wiki?
> >
> > Darren
> >
> >> On Oct 28, 2013, at 12:08 PM, Santhosh Edukulla <
> santhosh.edukulla@citrix.com> wrote:
> >>
> >> 1.It seems we already have a code coverage numbers using sonar as
> below. It currently shows only the numbers for unit tests.
> >>
> >> https://analysis.apache.org/dashboard/index/100206
> >>
> >> 2. The below link has an explanation for using it for both integration
> and unit tests.
> >>
> >>
> http://docs.codehaus.org/display/SONAR/Code+Coverage+by+Integration+Tests+for+Java+Project
> >>
> >> 3. Many links suggests it has good decision coverage facility compared
> to other coverage tools.
> >>
> >>
> http://onlysoftware.wordpress.com/2012/12/19/code-coverage-tools-jacoco-cobertura-emma-comparison-in-sonar/
> >>
> >> Regards,
> >> Santhosh
> >> ________________________________________
> >> From: Laszlo Hornyak [laszlo.hornyak@gmail.com]
> >> Sent: Monday, October 28, 2013 1:43 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: Re: Tiered Quality
> >>
> >> Sonar already tracks the unit test coverage. It is also able to track
> the
> >> integration test coverage, however this might be a bit more
> sophisticated
> >> in CS since not all hardware/software requirements are available in the
> >> jenkins environment. However, this could be a problem in any
> environment.
> >>
> >>
> >>> On Mon, Oct 28, 2013 at 5:53 AM, Prasanna Santhanam <tsp@apache.org>
> wrote:
> >>>
> >>> We need a way to check coverage of (unit+integration) tests. How many
> >>> lines of code hit on a deployed system that corresponds to the
> >>> component donated/committed. We don't have that for existing tests so
> >>> it makes it hard to judge if a feature that comes with tests covers
> >>> enough of itself.
> >>>
> >>>> On Sun, Oct 27, 2013 at 11:00:46PM +0100, Laszlo Hornyak wrote:
> >>>> Ok, makes sense, but that sounds like even more work :) Can you share
> the
> >>>> plan on how will this work?
> >>>>
> >>>>
> >>>> On Sun, Oct 27, 2013 at 7:54 PM, Darren Shepherd <
> >>>> darren.s.shepherd@gmail.com> wrote:
> >>>>
> >>>>> I think it can't be at a component level because components are
too
> >>> large.
> >>>>> It needs to be at a feature for implementation level.  For example,
> >>> live
> >>>>> storage migration for xen and live storage migration for kvm (don't
> >>> know if
> >>>>> that's a real thing) would be two separate items.
> >>>>>
> >>>>> Darren
> >>>>>
> >>>>>> On Oct 27, 2013, at 10:57 AM, Laszlo Hornyak <
> >>> laszlo.hornyak@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> I believe this will be very useful for users.
> >>>>>> As far as I understand someone will have to qualify components.
What
> >>> will
> >>>>>> be the method for qualification? I do not think simply the test
> >>> coverage
> >>>>>> would be right. But then if you want to go deeper, then you
need a
> >>> bigger
> >>>>>> effort testing the components.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Oct 27, 2013 at 4:51 PM, Darren Shepherd <
> >>>>>> darren.s.shepherd@gmail.com> wrote:
> >>>>>>
> >>>>>>> I don't know if a similar thing has been talked about before
but I
> >>>>>>> thought I'd just throws this out there.  The ultimate way
to ensure
> >>>>>>> quality is that we have unit test and integration test coverage
on
> >>> all
> >>>>>>> functionality.  That way somebody authors some code, commits
to,
> for
> >>>>>>> example, 4.2, but then when we release 4.3, 4.4, etc they
aren't on
> >>>>>>> the hook to manually tests the functionality with each release.
>  The
> >>>>>>> obvious nature of a community project is that people come
and go.
> >>> If
> >>>>>>> a contributor wants to ensure the long term viability of
the
> >>>>>>> component, they should ensure that there are unit+integration
> tests.
> >>>>>>>
> >>>>>>> Now, for whatever reason whether good or bad, it's not always
> >>> possible
> >>>>>>> to have full integration tests.  I don't want to throw down
the
> >>> gamut
> >>>>>>> and say everything must have coverage because that will
mean some
> >>>>>>> useful code/feature will not get in because of some coverage
wasn't
> >>>>>>> possible at the time.
> >>>>>>>
> >>>>>>> What I propose is that for every feature or function we
put it in a
> >>>>>>> tier of what is the quality of it (very similar to how OpenStack
> >>>>>>> qualifies their hypervisor integration).  Tier A means unit
test
> and
> >>>>>>> integration test coverage gates the release.  Tier B means
unit
> test
> >>>>>>> coverage gates the release.  Tier C mean who knows, it compiled.
>  We
> >>>>>>> can go through and classify the components and then as a
community
> >>> we
> >>>>>>> can try to get as much into Tier A as possible.
> >>>>>>>
> >>>>>>> Darren
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> EOF
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> EOF
> >>>
> >>> --
> >>> Prasanna.,
> >>>
> >>> ------------------------
> >>> Powered by BigRock.com
> >>
> >>
> >> --
> >>
> >> EOF
>



--

EOF
Mime
View raw message