lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <>
Subject Re: Lucene's default settings & back compatibility
Date Wed, 10 Jun 2009 16:32:59 GMT
No one really responded to this Shai? And I take it that the user list 
never saw it?

Perhaps we should just ask for opinion from the user list based on what 
you already have - just to gauge the reaction on different points. 
Unless someone responds shortly, we could take a year waiting to shake 
it out.
The threat of sending should prompt anyone with any issues to speak up.

I think we should add though:
explicitly what has changed (eg if we switch something, what was the 
policy before - most users won't even know)
an overview of why we are interested in relaxing back compat

- Mark

Shai Erera wrote:
> Ok, so digging back in this thread, I think the following proposals 
> were made (if I missed some, please add them):
> 1. API deprecation last *at least* one full minor release. Example: if 
> we deprecate an API in 2.4, we can remove it in 2.5. BUT, we are also 
> free to keep it there and remove it in 2.6, 2.9, 3.0, 3.5. I would 
> like to reserve that option for controversial deprecations, like 
> TokenStream, and maybe even the HitCollector recent changes. Those 
> that we feel will have a large impact on the users, we might want to 
> keep around for a bit longer until we get enough feedback from the 
> field and are more confident with that change.
> 2. Bugs are fixed backwards on the last "dot" release only. Example, A 
> bug that's discovered after 2.4 is released, is fixed on 2.4.X branch. 
> Once 2.5 is released, any bug fixes happen on trunk and 2.5.X. A 
> slight relaxation would be adding something like "we may still fix 
> bugs on the 2.4.X branch if we feel it's important enough". For 
> example if 2.5 contains a lot of API changes and we think a 
> considerable portion of our users are still on 2.4.
> 3. Jar drop-in ability is only guaranteed on point releases (this is 
> slightly of an outcome of (1) and (2), but (6) will also affect it).
> 4. Changes to the index format last at least one full major release. 
> Example: a change to the index format in 2.X, is supported in all 3.Y 
> releases, and removed in 4.0. Again, I write "at least" since we 
> should have the freedom to extend support for a particular change.
> 5. Changes to the default settings are allowed between minor releases, 
> provided that we give the users a way to revert back to the old 
> behavior. Examples are LUCENE-1542 and the latest issues Mike opened. 
> Those changes will be applied out-of-the-box. The provided API to 
> revert to the old behavior may be a supported API, or a deprecated 
> API. For deprecation we can decide to keep the API longer than one 
> minor release.
> 5.1) An exception to (5) are bug fixes which break back-compat - those 
> are always visible, w/ a way to revert to the buggy behavior. That way 
> may be deprecated or not, and its support lifetime can be made on a 
> case-by-case basis.
> 6. Minor changes to APIs can happen w/o any deprecation. Example, 
> LUCENE-1614, adding 1/2 methods to an interface with a good 
> documentation and trivial proposal for implementation etc.
> You will notice that almost every proposal has a "we may decide to 
> keep it for longer" - I wrote it following one of the early responses 
> on this thread (I think it was Grant's) - we should not attempt to set 
> things in stone. Our back-compat policy should ensure some level of 
> SLA to our users, but otherwise we should not act as robots, and if we 
> think a certain case requires a different handling than the policy 
> states (only for the user's benefit though), it should be done that 
> way. The burden is still put on the committers, only now the policy is 
> relaxed a bit, and handles different cases in different ways, and the 
> committers/contributors don't need to feel that their hands are tied.
> These set the ground/basis, but otherwise we should decide on a 
> case-by-case basis on any extension/relaxation of the policy, for our 
> users' benefits. After quite some time I've been following the 
> discussions on this mailing list, I don't remember ever seeing an 
> issue being driven against our users' benefit. All issues attempt to 
> improve Lucene's performance and our users' experience (end users as 
> well as search application developers). I think it's only fair to ask 
> this "users" community be more forgiving and open to make changes on 
> their side too, making the life of the committers/contributors a bit 
> easier.
> I also agree that the next step would be taking this to java-user and 
> get a sense of whether our "users" community agree with those changes 
> or not. I hope that the above summary captures what's needed to be 
> sent to this list.
> Shai
> On Sat, May 30, 2009 at 2:21 PM, Michael McCandless 
> < <>> wrote:
>     Actually, I think this is a common, and in fact natural/expected
>     occurrence in open-source.  When a tricky topic is discussed, and the
>     opinions are often divergent, frequently the conversation never
>     "converges" to a consensus and the discussion dies.  Only if
>     discussion reaches a semblance of consensus do we vote on it.
>     It's exactly like what happens when a controversial bill tries to go
>     through the US congress.  It's heavily discussed and then dies off
>     from lack of consensus, or, it gets far enough to be voted on.
>     Ie, this is completely normal for open source.
>     We may not like it, we may consider it inefficient, annoying,
>     frustrating, whatever, but this is in fact a reality of all healthy
>     open-source projects.
>     Consensus building is not easy, and if the number of people trying to
>     build consensus, by iterating on the proposal, compromising,
>     suggesting alternatives when others dislike an approach, etc., is
>     dwarfed by the number of people objecting to the proposal, then
>     consensus never emerges.
>     In this case specifically, I had a rather singular goal: the freedom
>     to make changes to defaults inside Lucene to always favor new users,
>     while not hurting back-compat users.  I intentionally proposed no
>     changes to our back-compat policy (knowing reaching consensus would be
>     that much more difficult).
>     The proposal went through several iterations (*settings,
>     *actsAsVersion, etc) that all failed to reach consensus, so we settled
>     back on the current approach of "make the setting explicit" which is
>     an OK workaround.  One by one I've been doing that for the original
>     examples I listed (readOnly IndexReader, NIOFSDir default, etc.)
>     But, then, the conversation shifted to a different topic ("how to
>     relax our back-compat policy"), which also failed to reach consensus.
>     Maybe, the best way forward is to break out each of the separate
>     bullets and discuss them separately?
>     Mike
>     On Fri, May 29, 2009 at 11:22 PM, Shai Erera <
>     <>> wrote:
>     > So ... I've this happen a lot of times (especially in my thesis
>     work) -
>     > someone raises a controversial topic, or one that touches the
>     nervous of the
>     > system, there's a flurry of activity and then it dies
>     unexpectedly, even
>     > though it feels to everyone that there's "an extra mile" that
>     should be
>     > taken in order to bring it to completion.
>     >
>     > And that's what I've seen in this thread. A lot has been said -
>     lots of
>     > comments, ideas, opinions. Lots of ranting and complaining. Then
>     it died ...
>     > Thank you Grant for that last "beep", I hope that was an
>     intention to
>     > resurrect it.
>     >
>     > So I ask - how come that we don't have a decision? Is it because
>     we're
>     > "afraid" to make a decision? (that last sentence is supposed to
>     "tease" the
>     > community, not to pass judgement)
>     >
>     > I'm asking because it seems like everybody pretty much agrees on
>     most of the
>     > suggestions, so why not decide "let's do X, Y and Z" and change the
>     > back-compat page starting from 2.9? If people don't remember the
>     decisions,
>     > I don't mind reiterating them.
>     >
>     > (I also ask because I'd like to take the improvements from
>     LUCENE-1614 to
>     > TermDocs/Positions, PhrasePositions, Spans. All except
>     PhrasePositions are
>     > public interfaces and so it matters if I need to go through creating
>     > abstract classes, with new names, or I can change those
>     interfaces, asking
>     > those that implemented their own TermDocs to modify the code).
>     >
>     > Shai
>     >
>     > On Wed, May 27, 2009 at 10:36 PM, Grant Ingersoll
>     < <>>
>     > wrote:
>     >>
>     >> So, here's a real, concrete example of the need for case by
>     case back
>     >> compat.  See
>     >>
>     >> It's completely stupid that ExtendedFieldCache even exists.  
>     It is a dumb
>     >> workaround for a made up problem that has nothing to do with
>     real coders
>     >> living in the modern age of development where IDE's make
>     refactoring these
>     >> types of things very cheap.  Namely, the notion that interfaces
>     must never
>     >> change lest every 6-9 months some minute number of users (I'd
>     venture it's
>     >> less than 1% of users) out there, who by any account are
>     completely capable
>     >> of implementing hard core Lucene internals (like extending
>     FieldCache), yet
>     >> are seemingly incapable of reading a CHANGES file with a huge
>     disclaimer in
>     >> it, have to recompile (GASP!) their code and put in a dummy
>     implementation
>     >> of some new interface method.  Yet, here we are with Yonik
>     fixing very real
>     >> problems that are a direct result of coding around back compat.
>     (along with
>     >> a mistake; it took a long time for this issue to even be
>     discovered) that
>     >> very much effect the usability of Lucene and the day to day
>     experience of a
>     >> good number of users.
>     >>
>     >> In other words, the real fix for L-1662 is for ExtFieldCache to
>     be folded
>     >> into FieldCache and for the file to be removed, never to be
>     heard from
>     >> again.
>     >>
>     >> The same can be said for the whole Fieldable issue, but that's
>     a different
>     >> day.
>     >>
>     >> Ranting,
>     >> Grant
>     >>
>     >>
>     ---------------------------------------------------------------------
>     >> To unsubscribe, e-mail:
>     <>
>     >> For additional commands, e-mail:
>     <>
>     >>
>     >
>     >
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail:
>     <>
>     For additional commands, e-mail:
>     <>

- Mark

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message