lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <karl.wri...@nokia.com>
Subject RE: Brainstorming on Improving the Release Process
Date Wed, 30 Mar 2011 12:46:46 GMT
Hi Grant,

This is a great post.

I'm not a committer for Lucene or Solr, but I'm seriously thinking that much of what Lucene/Solr
does right should be considered by the project I AM a committer for: ManifoldCF.

Key things I would add based on experience with commercial software development:

(A) Left to their own devices, releases almost always get "too big".  The temptation is to
just keep adding stuff, which winds up causing a delay, which adds more pressure for more
features to be added to the release, etc.  The only way to address this that I've found which
works is the "train leaving the station" model, where the date gets set in advance, and decisions
as to feature inclusion based on that date.  Practically speaking, this means extended periods
of time where development is happening in trunk and only selected changes are being pulled
up to the release branch.

(B) Corollary to the "train leaving the station" model is that any massive global changes
must occur only towards the beginning of the cycle.  Changes added to the release later in
the cycle must be less and less destabilizing.  This often involves significant tradeoffs
of the "proper" way to do things vs. the "least risky" way to do things.

(C) Finally, the larger the release, the LONGER the release branch must be active.  If you
intend to release 4.0 this year, you should probably create the release branch no later than
May/June, given the size of the 4.0 release already.

I'm sure all of this is well known, but I thought I'd state it nonetheless.

Karl


-----Original Message-----
From: ext Grant Ingersoll [mailto:gsingers@apache.org] 
Sent: Wednesday, March 30, 2011 8:22 AM
To: dev@lucene.apache.org
Subject: Brainstorming on Improving the Release Process

(Long post, please bear with me and please read!)

Now that we have the release done (I'm working through the publication process now), I want
to start the process of thinking about how we can improve the release process.  As I see it,
building the artifacts and checking the legal items are now almost completely automated and
testable at earlier stages in the game. 

We have kept saying we want to release more often, but we have never defined actionable steps
with which we can get there.  Goals without actionable steps are useless.

So, with that in mind, I'd like to brainstorm on how we can improve things a bit more.  Several
us acted as RM this time around, so I think we have some common, shared knowledge to take
advantage of this time as opposed to in the past where one person mostly just did the release
in the background and then we all voted.

So, let's start with what we have right:

1. The Ant process for building a release candidate for both Lucene and Solr is almost identical
now and fairly straightforward.
2. I think the feature freeze is a good thing, although it is a bit too long perhaps.
3. Pretty good documentation on the steps involved to branch, etc.
4. The new license validation stuff is a start for enforcing licensing up front more effectively.
 What else can we validate up front in terms of packaging?
5. We have an awesome test infrastructure now.  I think it is safe to say that this version
of Lucene is easily the most tested version we have ever shipped.

Things I see that can be improved, and these are only suggestions:

1.  We need to define the Minimum Effective Dose (MED - http://gizmodo.com/#!5709902/4+hour-body-the-principle-of-the-minimum-effective-dose)
for producing a quality release.  Nothing more, nothing less.  I think one of our biggest
problems is we don't know when we are done.  It's this loosey-goosey "we all agree" notion,
but that's silly.  It's software, we should be able to test almost all of the artifacts for
certain attributes and then release when they pass.  If we get something wrong, put in a test
for it in the next release.  The old saying about perfect being the enemy of great applies
here.

In other words, we don't have well defined things that we all are looking for when vetting
a release candidate, other than what the ASF requires.  Look at the last few vote threads
or for any of the previous threads.  It's obvious that we have a large variety of people doing
a large variety of things when it comes to testing the candidates.  For instance, I do the
following:
  a. check sigs., md5 hashes, etc. 
  b. run the demos, 
  c. run the Solr example and index some content, 
  d. check over the LICENSE, NOTICE, CHANGES files
  e. Check the overall packaging, etc. is reasonable
  f. I run them through my training code

Others clearly do many other things.  Many of you have your own benchmark tests you run, others
read over every last bit of documentation others still put the RC into their own application
and test it.   All of this is good, but the problem is it is not _shared_ until the actual
RC is up and it is not repeatable (not that all of it can be).  If you have benchmark code/tests
that your run on an RC that doesn't involve proprietary code, why isn't it donated to the
project so that we can all use it?  That way we don't have to wait until your -1 at the 11th
hour to realize the RC is not good.  I personally don't care whether it's python or perl or
whatever.  Something that works is better than nothing.  For instance, right now some of the
committers have an Apache Extras project going for benchmarking.  Can we get this running
on ASF resources on a regular basis?  If it's a computing resource issue, let's go to Infrastructure
and ask for resources.  Infrastructure has repeatedly said that if a project needs resources
to put together a proposal of what you want.  I bet we could get budget to spin up an EC2
instance once a week, run those long running tests (Test2B and other benchmarks) and then
report back.  All of that can be automated.

Also, please think hard about whether the things you test can be automated and built into
our test suite or at least run nightly or something on Jenkins and then donating them.  I
know reading documentation can't, but what else?  For instance, could we auto-generate the
file formats documentation?

2. We should be running and testing the release packaging process more regularly.

3.  I had an epiphany this release and it came via Hoss on a non release related issue where,
likely unbeknownst to him, he called me out for not being focused on the release (https://issues.apache.org/jira/browse/SOLR-2366?focusedCommentId=12996154&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12996154)
 I have often been a passive observer when it comes to releasing other than testing the RC
(either that or I am the RM, it seems)  It's easy to sit and complain about there not being
more frequent releases, but what are you doing about it?  Why are we working on trunk when
we are trying to get a release out?  Obviously you are all free to do as you wish, but then
don't complain when there aren't more frequent releases.  Yes, I am calling you out just like
Hoss called me out.  If you aren't sure how you can help, ask.  Assuming we can better document
#1 above, helping should be a lot easier in the future.

Now, obviously, not everyone can be involved in the actual building of the RC, but that doesn't
mean you can't go build it yourself and start testing on it before the Vote comes out.  And
it also doesn't mean you can't take a hard look at the process and see if there aren't better
ways to do it.  I also think if we all focus more and coordinate our testing by defining test
plans up front, we could get releases out in far less time.

4. Are those last minute changes really blockers?  We are all perfectionists, I know.  Our
guilt gets to us if we happen to know there is something "broken" and we don't fix it.  I
think we should add a "Known Issues" section to CHANGES.txt and we should be a bit tougher
about keeping out last minute changes.   I was also Guilty as Charged in this case on the
highlighter issue that Mark and I were working on.  Thankfully, Robert set me right.  I think
that if it is only likely to effect a very small percentage of users AND there is a workaround,
then we should simply document it and fix it on trunk and in the branch but not in the release.
  If there are too many of these, then we are not ready to release.

5.  Most of the steps in http://wiki.apache.org/lucene-java/ReleaseTodo under the Branching
& Feature Freeze section can be automated.  Same with the Publication section.

So, what are some possible actionable steps we can take?

1. To address #1, we can publish a test plan just like you would do in your day job.  See
what OpenNLP does: https://cwiki.apache.org/confluence/display/OPENNLP/TestPlan1.5.1.  It
starts by everyone addressing #1 above on our Wiki.  See http://wiki.apache.org/lucene-java/TestPlans
and http://wiki.apache.org/solr/TestPlans for a starting spot.

2. To address #2, we can setup a Jenkins job that runs the packaging process regularly and
somehow figure out how to reliably test it.	

3. Shrugs on #3.  No one can make you participate, but we can make you feel guilty just like
Hoss made me feel guilty.  For PMC members specifically, however, _perhaps_ if you don't vote
for two or three releases in a row, then you shouldn't be on the PMC.  Your job on the PMC
primarily consists of two things: vote on committers, vote on releases.  I can see giving
a pass once given life happens.  Two or three in a row means you likely consider yourself
emeritus or you just like having the title but not having any of the responsibility.   At
a minimum send in a +0 and let us know you have a pulse.

4. For #4, this is a collective commitment to make the release happen as quickly as possible
while still being correct for most people in most situations.  Focus on the goal.

5. For #5, we just need some people with some Ant skills to make it happen.

6. If we want to release every 6 months, then let's setup some sort of reminder system that
nags us as we get closer or something.  Let's also be honest with ourselves about what features
are in the release and what are not.  We don't have to have jam everything in all the time.

OK, enough of my ideas.  And I do stress that they are only ideas/observations.  What do others
think?  Several of you have lived the RM pain now, let's take a good hard look at what can
be done better so that we can release more often.

-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message