lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Re: Brainstorming on Improving the Release Process
Date Wed, 30 Mar 2011 12:43:45 GMT
I'll also add:

We need to figure out a better approach for CHANGES.txt.  Step 4 of the Publishing process
is a PITA.

On Mar 30, 2011, at 8:22 AM, Grant Ingersoll wrote:

> (Long post, please bear with me and please read!)
> Now that we have the release done (I'm working through the publication process now),
I want to start the process of thinking about how we can improve the release process.  As
I see it, building the artifacts and checking the legal items are now almost completely automated
and testable at earlier stages in the game. 
> We have kept saying we want to release more often, but we have never defined actionable
steps with which we can get there.  Goals without actionable steps are useless.
> So, with that in mind, I'd like to brainstorm on how we can improve things a bit more.
 Several us acted as RM this time around, so I think we have some common, shared knowledge
to take advantage of this time as opposed to in the past where one person mostly just did
the release in the background and then we all voted.
> So, let's start with what we have right:
> 1. The Ant process for building a release candidate for both Lucene and Solr is almost
identical now and fairly straightforward.
> 2. I think the feature freeze is a good thing, although it is a bit too long perhaps.
> 3. Pretty good documentation on the steps involved to branch, etc.
> 4. The new license validation stuff is a start for enforcing licensing up front more
effectively.  What else can we validate up front in terms of packaging?
> 5. We have an awesome test infrastructure now.  I think it is safe to say that this version
of Lucene is easily the most tested version we have ever shipped.
> Things I see that can be improved, and these are only suggestions:
> 1.  We need to define the Minimum Effective Dose (MED -!5709902/4+hour-body-the-principle-of-the-minimum-effective-dose)
for producing a quality release.  Nothing more, nothing less.  I think one of our biggest
problems is we don't know when we are done.  It's this loosey-goosey "we all agree" notion,
but that's silly.  It's software, we should be able to test almost all of the artifacts for
certain attributes and then release when they pass.  If we get something wrong, put in a test
for it in the next release.  The old saying about perfect being the enemy of great applies
> In other words, we don't have well defined things that we all are looking for when vetting
a release candidate, other than what the ASF requires.  Look at the last few vote threads
or for any of the previous threads.  It's obvious that we have a large variety of people doing
a large variety of things when it comes to testing the candidates.  For instance, I do the
>  a. check sigs., md5 hashes, etc. 
>  b. run the demos, 
>  c. run the Solr example and index some content, 
>  d. check over the LICENSE, NOTICE, CHANGES files
>  e. Check the overall packaging, etc. is reasonable
>  f. I run them through my training code
> Others clearly do many other things.  Many of you have your own benchmark tests you run,
others read over every last bit of documentation others still put the RC into their own application
and test it.   All of this is good, but the problem is it is not _shared_ until the actual
RC is up and it is not repeatable (not that all of it can be).  If you have benchmark code/tests
that your run on an RC that doesn't involve proprietary code, why isn't it donated to the
project so that we can all use it?  That way we don't have to wait until your -1 at the 11th
hour to realize the RC is not good.  I personally don't care whether it's python or perl or
whatever.  Something that works is better than nothing.  For instance, right now some of the
committers have an Apache Extras project going for benchmarking.  Can we get this running
on ASF resources on a regular basis?  If it's a computing resource issue, let's go to Infrastructure
and ask for resources.  Infrastructure has repeatedly said that if a project needs resources
to put together a proposal of what you want.  I bet we could get budget to spin up an EC2
instance once a week, run those long running tests (Test2B and other benchmarks) and then
report back.  All of that can be automated.
> Also, please think hard about whether the things you test can be automated and built
into our test suite or at least run nightly or something on Jenkins and then donating them.
 I know reading documentation can't, but what else?  For instance, could we auto-generate
the file formats documentation?
> 2. We should be running and testing the release packaging process more regularly.
> 3.  I had an epiphany this release and it came via Hoss on a non release related issue
where, likely unbeknownst to him, he called me out for not being focused on the release (
 I have often been a passive observer when it comes to releasing other than testing the RC
(either that or I am the RM, it seems)  It's easy to sit and complain about there not being
more frequent releases, but what are you doing about it?  Why are we working on trunk when
we are trying to get a release out?  Obviously you are all free to do as you wish, but then
don't complain when there aren't more frequent releases.  Yes, I am calling you out just like
Hoss called me out.  If you aren't sure how you can help, ask.  Assuming we can better document
#1 above, helping should be a lot easier in the future.
> Now, obviously, not everyone can be involved in the actual building of the RC, but that
doesn't mean you can't go build it yourself and start testing on it before the Vote comes
out.  And it also doesn't mean you can't take a hard look at the process and see if there
aren't better ways to do it.  I also think if we all focus more and coordinate our testing
by defining test plans up front, we could get releases out in far less time.
> 4. Are those last minute changes really blockers?  We are all perfectionists, I know.
 Our guilt gets to us if we happen to know there is something "broken" and we don't fix it.
 I think we should add a "Known Issues" section to CHANGES.txt and we should be a bit tougher
about keeping out last minute changes.   I was also Guilty as Charged in this case on the
highlighter issue that Mark and I were working on.  Thankfully, Robert set me right.  I think
that if it is only likely to effect a very small percentage of users AND there is a workaround,
then we should simply document it and fix it on trunk and in the branch but not in the release.
  If there are too many of these, then we are not ready to release.
> 5.  Most of the steps in under the Branching
& Feature Freeze section can be automated.  Same with the Publication section.
> So, what are some possible actionable steps we can take?
> 1. To address #1, we can publish a test plan just like you would do in your day job.
 See what OpenNLP does:
 It starts by everyone addressing #1 above on our Wiki.  See
and for a starting spot.
> 2. To address #2, we can setup a Jenkins job that runs the packaging process regularly
and somehow figure out how to reliably test it.	
> 3. Shrugs on #3.  No one can make you participate, but we can make you feel guilty just
like Hoss made me feel guilty.  For PMC members specifically, however, _perhaps_ if you don't
vote for two or three releases in a row, then you shouldn't be on the PMC.  Your job on the
PMC primarily consists of two things: vote on committers, vote on releases.  I can see giving
a pass once given life happens.  Two or three in a row means you likely consider yourself
emeritus or you just like having the title but not having any of the responsibility.   At
a minimum send in a +0 and let us know you have a pulse.
> 4. For #4, this is a collective commitment to make the release happen as quickly as possible
while still being correct for most people in most situations.  Focus on the goal.
> 5. For #5, we just need some people with some Ant skills to make it happen.
> 6. If we want to release every 6 months, then let's setup some sort of reminder system
that nags us as we get closer or something.  Let's also be honest with ourselves about what
features are in the release and what are not.  We don't have to have jam everything in all
the time.
> OK, enough of my ideas.  And I do stress that they are only ideas/observations.  What
do others think?  Several of you have lived the RM pain now, let's take a good hard look at
what can be done better so that we can release more often.
> -Grant

Grant Ingersoll

Search the Lucene ecosystem docs using Solr/Lucene:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message