cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@gmail.com>
Subject Re: Tiered Quality
Date Sun, 03 Nov 2013 12:05:38 GMT
You are right Laszlo but i was reacting on the scoring of submissions
proposal. if the goal is getting 3.6 to 3.7 coverage we definitely
don't need anything but a keen eye on submissions.

On Fri, Nov 1, 2013 at 6:48 PM, Laszlo Hornyak <laszlo.hornyak@gmail.com> wrote:
> I have heard about commercial tools that do more advanced coverage
> tracking. But if you think in open source, not sure Sonar really has an
> alternative. It is pretty cool anyway.
> Btw the overall code coverage is about 3.6%, probaly it is not worth trying
> something more advanced for that much.
>
>
> On Thu, Oct 31, 2013 at 9:12 PM, Daan Hoogland <daan.hoogland@gmail.com>wrote:
>
>> one note on testing guys,
>>
>> I see that the analysis site give lines-coverage and branch-coverage.
>> I don't see anything on distinct paths. What I mean is that the
>> program
>> if(a)
>>  A
>> else
>>  B
>> if(b)
>>  C
>> else
>>  D
>> if(c)
>>  E
>> else
>>  F
>> has eight (2^3) distict paths. It is not enough to show that
>> A,B,C,D,E,F are all hit and hance every line and branch. Also all
>> combinations of a/!a and b/!b and c/!c need to be hit.
>>
>> Now I am not saying that we should not score our code if not in this
>> way but it is kind of kidding ourselves if we don't face up to the
>> fact that coverage of lines of code or branches is not a completeness
>> criterium of some kind. I don't know if any of the mentioned tools
>> does analysis this thorough. But if any does we should go for that
>> one.
>>
>> €0,02
>> Daan
>>
>> On Tue, Oct 29, 2013 at 2:21 AM, Darren Shepherd
>> <darren.s.shepherd@gmail.com> wrote:
>> > Starting with the honor system might be good.  It's not so easy some
>> times to relate lines of code to functionality.  Also just because it hits
>> a line of code doesn't mean it's really tested.
>> >
>> > Can't we just get people to just put a check mark on some table in the
>> wiki?
>> >
>> > Darren
>> >
>> >> On Oct 28, 2013, at 12:08 PM, Santhosh Edukulla <
>> santhosh.edukulla@citrix.com> wrote:
>> >>
>> >> 1.It seems we already have a code coverage numbers using sonar as
>> below. It currently shows only the numbers for unit tests.
>> >>
>> >> https://analysis.apache.org/dashboard/index/100206
>> >>
>> >> 2. The below link has an explanation for using it for both integration
>> and unit tests.
>> >>
>> >>
>> http://docs.codehaus.org/display/SONAR/Code+Coverage+by+Integration+Tests+for+Java+Project
>> >>
>> >> 3. Many links suggests it has good decision coverage facility compared
>> to other coverage tools.
>> >>
>> >>
>> http://onlysoftware.wordpress.com/2012/12/19/code-coverage-tools-jacoco-cobertura-emma-comparison-in-sonar/
>> >>
>> >> Regards,
>> >> Santhosh
>> >> ________________________________________
>> >> From: Laszlo Hornyak [laszlo.hornyak@gmail.com]
>> >> Sent: Monday, October 28, 2013 1:43 PM
>> >> To: dev@cloudstack.apache.org
>> >> Subject: Re: Tiered Quality
>> >>
>> >> Sonar already tracks the unit test coverage. It is also able to track
>> the
>> >> integration test coverage, however this might be a bit more
>> sophisticated
>> >> in CS since not all hardware/software requirements are available in the
>> >> jenkins environment. However, this could be a problem in any
>> environment.
>> >>
>> >>
>> >>> On Mon, Oct 28, 2013 at 5:53 AM, Prasanna Santhanam <tsp@apache.org>
>> wrote:
>> >>>
>> >>> We need a way to check coverage of (unit+integration) tests. How many
>> >>> lines of code hit on a deployed system that corresponds to the
>> >>> component donated/committed. We don't have that for existing tests so
>> >>> it makes it hard to judge if a feature that comes with tests covers
>> >>> enough of itself.
>> >>>
>> >>>> On Sun, Oct 27, 2013 at 11:00:46PM +0100, Laszlo Hornyak wrote:
>> >>>> Ok, makes sense, but that sounds like even more work :) Can you
share
>> the
>> >>>> plan on how will this work?
>> >>>>
>> >>>>
>> >>>> On Sun, Oct 27, 2013 at 7:54 PM, Darren Shepherd <
>> >>>> darren.s.shepherd@gmail.com> wrote:
>> >>>>
>> >>>>> I think it can't be at a component level because components
are too
>> >>> large.
>> >>>>> It needs to be at a feature for implementation level.  For example,
>> >>> live
>> >>>>> storage migration for xen and live storage migration for kvm
(don't
>> >>> know if
>> >>>>> that's a real thing) would be two separate items.
>> >>>>>
>> >>>>> Darren
>> >>>>>
>> >>>>>> On Oct 27, 2013, at 10:57 AM, Laszlo Hornyak <
>> >>> laszlo.hornyak@gmail.com>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> I believe this will be very useful for users.
>> >>>>>> As far as I understand someone will have to qualify components.
What
>> >>> will
>> >>>>>> be the method for qualification? I do not think simply the
test
>> >>> coverage
>> >>>>>> would be right. But then if you want to go deeper, then
you need a
>> >>> bigger
>> >>>>>> effort testing the components.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sun, Oct 27, 2013 at 4:51 PM, Darren Shepherd <
>> >>>>>> darren.s.shepherd@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> I don't know if a similar thing has been talked about
before but I
>> >>>>>>> thought I'd just throws this out there.  The ultimate
way to ensure
>> >>>>>>> quality is that we have unit test and integration test
coverage on
>> >>> all
>> >>>>>>> functionality.  That way somebody authors some code,
commits to,
>> for
>> >>>>>>> example, 4.2, but then when we release 4.3, 4.4, etc
they aren't on
>> >>>>>>> the hook to manually tests the functionality with each
release.
>>  The
>> >>>>>>> obvious nature of a community project is that people
come and go.
>> >>> If
>> >>>>>>> a contributor wants to ensure the long term viability
of the
>> >>>>>>> component, they should ensure that there are unit+integration
>> tests.
>> >>>>>>>
>> >>>>>>> Now, for whatever reason whether good or bad, it's not
always
>> >>> possible
>> >>>>>>> to have full integration tests.  I don't want to throw
down the
>> >>> gamut
>> >>>>>>> and say everything must have coverage because that will
mean some
>> >>>>>>> useful code/feature will not get in because of some
coverage wasn't
>> >>>>>>> possible at the time.
>> >>>>>>>
>> >>>>>>> What I propose is that for every feature or function
we put it in a
>> >>>>>>> tier of what is the quality of it (very similar to how
OpenStack
>> >>>>>>> qualifies their hypervisor integration).  Tier A means
unit test
>> and
>> >>>>>>> integration test coverage gates the release.  Tier B
means unit
>> test
>> >>>>>>> coverage gates the release.  Tier C mean who knows,
it compiled.
>>  We
>> >>>>>>> can go through and classify the components and then
as a community
>> >>> we
>> >>>>>>> can try to get as much into Tier A as possible.
>> >>>>>>>
>> >>>>>>> Darren
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> EOF
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>>
>> >>>> EOF
>> >>>
>> >>> --
>> >>> Prasanna.,
>> >>>
>> >>> ------------------------
>> >>> Powered by BigRock.com
>> >>
>> >>
>> >> --
>> >>
>> >> EOF
>>
>
>
>
> --
>
> EOF

Mime
View raw message