lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Proposal about Version API "relaxation"
Date Thu, 15 Apr 2010 19:50:44 GMT
I wish we could have a face to face talk like in the evenings at ApacheCon :(

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Grant Ingersoll [mailto:gsiasf@gmail.com] On Behalf Of Grant
> Ingersoll
> Sent: Thursday, April 15, 2010 9:46 PM
> To: java-dev@lucene.apache.org
> Subject: Re: Proposal about Version API "relaxation"
> 
> From IRC:
> "why do I get the feeling that everyone is in "heated agreement" on the
> Version thread?
> there are some cases that mean people will have to reindex
> in those cases, we should tell people they will have to reindex
> then they can decide to upgrade or not
> all other cases, just do the sensible thing and test first
> I have yet to meet anyone who simply drops a new version into
> production and says go"
> 
> So, as I said earlier, why don't we just move forward with it, strive
> to support reading X-1 index format in X and let the user know the
> cases in which they will have to re-index. If a migration tool is
> necessary, then someone can write it at the appropriate time.  Just as
> was said w/ the Solr merge, it's software.  If it doesn't work, we can
> change it.  Thank goodness we don't have a back compatibility policy
> for our policies!
> 
> -Grant
> 
> 
> 
> 
> On Apr 15, 2010, at 3:35 PM, Michael McCandless wrote:
> 
> > Unfortunately, live searching against an old index can get very
> hairy.
> > EG look at what I had to do for the "flex API on pre-flex index" flex
> > emulation layer.
> >
> > It's also not great because it gives the illusion that all is good,
> > yet, you've taken a silent hit (up to ~10% or so) in your search
> > perf.
> >
> > Whereas building & maintaining a one-time index migration tool, in
> > contrast, is much less work.
> >
> > I realize the migration tool has issues -- it fixes the hard changes
> > but silently allows the soft changes to break (ie, your analyzers my
> > not produce the same tokens, until we move all core analyzers outside
> > of core, so they are separately versioned), but it seems like a good
> > compromise here?
> >
> > Mike
> >
> > 2010/4/15 Shai Erera <serera@gmail.com>:
> >> The reason Earwin why online migration is faster is because when u
> >> finally need to *fully* migrate your index, most chances are that
> most
> >> of the segments are already on the newer format. Offline migration
> >> will just keep the application idle for some amount of time until
> ALL
> >> segments are migrated.
> >>
> >> During the lifecycle of the index, segments are merged anyway, so
> >> migrating them on the fly virtually costs nothing. At the end, when
> u
> >> upgrade to a Lucene version which doesn't support the previous index
> >> format, you'll on the worse case need to migrate few large segments
> >> which were never merged. I don't know how many of those there will
> be
> >> as it really depends on the application, but I'd bet this process
> will
> >> touch just a few segments. And hence, throughput wise it will be a
> lot
> >> faster.
> >>
> >> We should create a migrate() API on IW which will touch just those
> >> segments and not incur a full optimize. That API can also be used
> for
> >> an offline migration tool, if we decide that's what we want.
> >>
> >> Shai
> >>
> >> On Thursday, April 15, 2010, jm <jmuguruza@gmail.com> wrote:
> >>> Not sure if plain users are allowed/encouraged to post in this
> list,
> >>> but wanted to mention (just an opinion from a happy user), as other
> >>> users have, that not all of us can reindex just like that. It would
> >>> not be 10 min for one of our installations for sure...
> >>>
> >>> First, i would need to implement some code to reindex, cause my
> source
> >>> data is postprocessed/compressed/encrypted/moved after it arrives
> to
> >>> the application, so I would need to retrieve all etc. And then
> >>> reindexing it would take days.
> >>> javier
> >>>
> >>> On Thu, Apr 15, 2010 at 9:04 PM, Earwin Burrfoot <earwin@gmail.com>
> wrote:
> >>>>> BTW Earwin, we can come up w/ a migrate() method on IW to
> accomplish
> >>>>> manual migration on the segments that are still on old versions.
> >>>>> That's not the point about whether optimize() is good or not. It
> is
> >>>>> the difference between telling the customer to run a 5-day
> migration
> >>>>> process, or a couple of hours. At the end of the day, the same
> >>>>> migration code will need to be written whether for the manual or
> >>>>> automatic case. And probably by the same developer which changed
> the
> >>>>> index format. It's the difference of when does it happen.
> >>>>
> >>>> Converting stuff is easier then emulating, that's exactly why I
> want a
> >>>> separate tool.
> >>>> There's no need to support cross-version merging, nor to emulate
> old APIs.
> >>>>
> >>>> I also don't understand why offline migration is going to take
> days
> >>>> instead of hours for online migration??
> >>>> WTF, it's gonna be even faster, as it doesn't have to merge
> things.
> >>>>
> >>>> --
> >>>> Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
> >>>> Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
> >>>> ICQ: 104465785
> >>>>
> >>>> ------------------------------------------------------------------
> ---
> >>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>>
> >>>>
> >>>
> >>> -------------------------------------------------------------------
> --
> >>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>>
> >>>
> >>
> >> --------------------------------------------------------------------
> -
> >> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-dev-help@lucene.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> 
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
> 
> Search the Lucene ecosystem using Solr/Lucene:
> http://www.lucidimagination.com/search
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message