gump-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From j...@apache.org
Subject [jira] Commented: (GUMP-21) FOG factor is misleading and requires improvement
Date Fri, 05 Mar 2004 01:48:32 GMT
The following comment has been added to this issue:

     Author: Daniel Savarese
    Created: Thu, 4 Mar 2004 5:47 PM
       Body:
Priority should be Minor.  I didn't notice it and let the default
go through.  Further discussion should probably occur on the Gump
mailing list.  Gump is probably going to start being used a lot
outside of the ASF this year, so I wanted to start the ball rolling
about nailing down what FOG is supposed to mean and figuring out
how to make it say better what it's supposed to mean.  I don't
think anyone knows the right answer right now, although we each
understand how to interpret independent metrics like commit
frequency, mean time between failures, mean time to resolve a
failure, number of dependent projects, etc.  There are many
things that are difficult to capture, such as projects that are
highly unstable during development, yet stabilize quickly as
a release target approaches.  My suspicion is that multiple
graphs of different behaviors will probably be more useful than
a single numerical factor.  In any case, this is not a pressing
issue at the moment, which is why I wanted to ask for this report
to be downgraded in priority.

---------------------------------------------------------------------
View this comment:
  http://nagoya.apache.org/jira/secure/ViewIssue.jspa?key=GUMP-21&page=comments#action_20970

---------------------------------------------------------------------
View the issue:
  http://nagoya.apache.org/jira/secure/ViewIssue.jspa?key=GUMP-21

Here is an overview of the issue:
---------------------------------------------------------------------
        Key: GUMP-21
    Summary: FOG factor is misleading and requires improvement
       Type: Improvement

     Status: Unassigned
   Priority: Major

    Project: Gump
 Components: 
             Python

   Assignee: 
   Reporter: Daniel Savarese

    Created: Thu, 4 Mar 2004 5:21 PM
    Updated: Thu, 4 Mar 2004 5:47 PM

Description:
FOG is defined right now as:
def getFOGFactor(self):
    return round((float(self.successes) / (float(self.failures) + float(self.prereqs))), 2)

This is not helpful.  Last month, jakart-oro
failed to build by coincidence around the time of a Gump
run and the problem was fixed before the next Gump run,
yet its FOG factor dropped in half.  The responsiveness
to resolving build and integration problems is not taken
into account.  It is likely that it is impossible for FOG
to mean anything because the factors that determine
a project's "health" (to borrow the term used by Adam Jack
in an email to the commons-dev and gump lists) are largely
orthogonal.  FOG should probably be a vector and if you really
wanted a single number you could use the length of that vector.

One of the major problems with the current factor is that
a project's FOG factor continues to increase when no changes
have been made to the repository.  It's not meaningful to
count only successful builds towards improving the factor
because no changes may have been made to the code base.
Ideally, you count only builds that cause no failures after
a change to the code base.  FOG doesn't factor in how quickly
(or slowly) projects recover from build failures, and how often
changes to a code base cause failures (either in itself or in
other projects).  Right now, a project with few code changes
(say once a month) that causes a failure 50% (1 every 2 months)
of the time the code is changed with a mean time of failure
correction of 1 week is deemed more reliable (higher FOG factor)
than a project that makes frequent changes (say every day) to
its code base and causes a failure 10% of the time (1 every 10
days) with a mean time of failure correction of 1 day.  That's
because FOG is based almost entirely on consecutive successful
Gump builds, which is not useful.

Rather than propose a new meaningless way of computing FOG, I
suggest FOG stop being used temporarily.  It would be more
useful to break out and report all of the independently useful
metrics and through experience over time determine how each
impacts the overall notion of FOG and eventually derive a
way of condensing the information (perhaps a FOG factor,
perhaps a vector) in a useful manner.  Right now the state
of affairs is that someone decided FOG was a cool thing, so
it's being reported and people see this number and mistakenly
think a project with half the FOG factor of another is
less dependable and therefore avoid using it.  Alternatively
an explanation of what FOG is intended to compute and how
it is computed should be placed on Gump reports that list
FOG so that consumers of the information will not be misled.





---------------------------------------------------------------------
JIRA INFORMATION:
This message is automatically generated by JIRA.

If you think it was sent incorrectly contact one of the administrators:
   http://nagoya.apache.org/jira/secure/Administrators.jspa

If you want more information on JIRA, or have a bug to report see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@gump.apache.org
For additional commands, e-mail: general-help@gump.apache.org


Mime
View raw message