incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ross Gardler (MS OPEN TECH)" <Ross.Gard...@microsoft.com>
Subject RE: dashboarding incubator
Date Sun, 23 Nov 2014 20:41:45 GMT
Thanks for your roundup here (very useful). You are making it clear that this is something
you might want to spend time on - I'll try and answer your final question ("is there already
any argument to say that inevitably the answer of the proof of concept will be negative?").
The short answer is yes and no. Below I spend most of the time explaining the "no", if you
want the short version skip to the "yes" in the last para.

Speaking entirely personally, I have always argued against using numbers to judge the health
of a project (which is the only natural outcome from collecting such numbers). For an ASF
project it's not absolute activity that is important. It's the strength and behavior of the
community and its governance that is important. Metrics do not provide this information, and
indeed can detract from the community health issues.

For example, many years ago we had a podling that didn't graduate for a long time because
we had evolved into having a hard metric on what "diversity" meant. It did graduate in the
end and today is a thriving and productive TLP. The significant expansion of the community
didn't happen until after graduation, a time when newcomers feel it is safe to invest in the
project. We've since scrapped the diversity metric and reverted to the original intent of
requiring a project to behave in a way that is welcoming to diverse interests (and thus capable
of satisfying a diversity metric given time). My point is that while some metrics can provide
indicators of something that needs looking into we need to ensure that the metrics do not
become more important than community health. 

Personally, I do use metrics when evaluating a project, but I use ones that are readily available
already through a number of other services. These are not "official" or "sanctioned" and therefore
say nothing, from an ASF perspective, about the health of a project. The danger I see is that
providing official metrics a) provides a level of "authority" to the metrics which most newcomers
are ill-equipped to evaluate and b) could lead to shortcut rules like "the metrics must show
there is X level of diversity/activity/volume/foobar" replacing proper evaluation of the project
and its community.

In summary, I'm not against metrics per se, I'm cautious about them becoming more important
than they should be. I can imagine the tools being somewhat useful *internally* where we can
ensure that expectations are properly managed. I am very cautious about using such metrics
externally where they can be quoted out of context and/or misrepresented.

All that being said, sponsors are increasingly asking us for metrics. For this reason I'm
vary interested in cross-foundational statistics rather than statistics about specific projects.
That is if you were to roll up the data from across the projects into valuable data about
the foundation as a whole I can see real value with minimal risk. Sally, over on press@ is
currently looking at the kind of data that would be useful to report.

Ross

-----Original Message-----
From: Sergio Fernández [mailto:sergio.fernandez@salzburgresearch.at] 
Sent: Sunday, November 23, 2014 11:37 AM
To: general@incubator.apache.org
Subject: Re: dashboarding incubator

After awaiting for feedback about my proposal, I understand there are three different aspects
that should be discussed:

* Cost: As Ross pointed, the potential prize is important to evaluate a solution. Although
I'd love to use the professional services of the company, the toolkit is open/free software
and be freely used, which moves more attention to the next point.

* Infrastructure requirements: Specially in the case we decide to provide all by ourselves,
such service would have some infrastructure requirements that need to be studied, as David
correctly pointed.

* Technical proposition: In the end the first two aspect should not be critical if the proposition
brings some value, to the project-level, Incubator or ASF.

I really see strong arguments against the proposal regarding the first two aspects. The third
is not that easy, since I do not see how such metrics should be used for evaluating projects,
rather than just bringing some indicators.

Before taking the discussion to the next level, where costs and resources need to be evaluated,
I opened this discussion proposing my time and personal resources to provide a simple proof
of concept. Then we should have more arguments (how much resources are actually required,
how useful are the indicators the dashboard provides, etc...) to move the discussion to the
next level.

But of course I'd like to have the good pleasure before investing time. 
So I'd like to ask the following question: is there already any argument to say that inevitably
the answer of the proof of concept will be negative?

Cheers,

On 21/11/14 21:27, Ross Gardler (MS OPEN TECH) wrote:
> We already evaluated the Bitergia offering - it is expensive and does 
> not provide sufficient benefit for the money (don't get me started on 
> how metrics are not a good evaluator of open source code...)
>
> I fully agree with the comments below.
>
> Ross
>
> -----Original Message-----
> From: jan i [mailto:jani@apache.org]
> Sent: Friday, November 21, 2014 12:24 PM
> To: general@incubator.apache.org
> Subject: Re: dashboarding incubator
>
> On 21 November 2014 20:44, Bertrand Delacretaz 
> <bdelacretaz@apache.org>
> wrote:
>
>> Hi,
>>
>> On Fri, Nov 21, 2014 at 3:35 PM, David Nalley <david@gnsa.us> wrote:
>>> ...I am generally against us standing up our own service that does this.
>>> We've had a couple of these systems over the years. (pulse.a,o for 
>>> instance). It takes a non-trivial amount of work to setup and 
>>> maintain such a system, and invariably it falls apart....
>>
>> I agree, OTOH if someone wants to help third parties get the data 
>> that they need to implement such services externally that might be fine.
>>
> Having our own service will only marginally provide us with something better, and will
cost (in endeffect) contractor resources, so I agree with david.
>
> rgds
> jan i.
>
>
>>
>> -Bertrand
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

--
Sergio Fernández
Senior Researcher
Knowledge and Media Technologies
Salzburg Research Forschungsgesellschaft mbH Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria
T: +43 662 2288 318 | M: +43 660 2747 925 sergio.fernandez@salzburgresearch.at
http://www.salzburgresearch.at

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Mime
View raw message