lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weiwei Wang <ww.wang...@gmail.com>
Subject Re: Index file compatibility and a migration plan to lucene 3
Date Wed, 09 Dec 2009 13:11:07 GMT
Hi, Rob,
I read
http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats and
found no compatibility guarantee for IndexWriter between different version.

You can run your idea as a test and see the output.
If it doesn't work, i suggest you convert your index to new version as I
said in the last post.

You can develop a convert tool to do this job automatically(that what i have
done).

If you do not have full access to the data center, you can read(readonly
mode is preferred) from the data center(through nfs or something like that)
and write to your local disk.

When all converting is done, you can copy the new index to the data center
with the help of the administrator.

On Wed, Dec 9, 2009 at 8:42 PM, Rob Staveley (Tom) <rstaveley@seseit.com>wrote:

> Thanks for the swift response, Weiwei.
>
> In my deployment, my index readers are in a data centre and therefore more
> difficult to upgrade than the writers. That's why I wanted to start with the
> writers rather than the readers. I realise that it looks the wrong way round
> and http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formatseffectively
says that changing the reader first is a better idea for most
> situations, but I wanted to know if writer first would work for me for 2.3.1
> -> 3.0.0.
>
> -----Original Message-----
> From: Weiwei Wang [mailto:ww.wang.cs@gmail.com]
> Sent: 09 December 2009 12:21
> To: java-user@lucene.apache.org
> Subject: Re: Index file compatibility and a migration plan to lucene 3
>
> I’ve finished a upgrade from 2.4.1 to 3.0.0
>
> What I do is like this:
> 1. Upgrade my user-defined analyzer, tokenizer and filter to 3.0.0
> 2. Use a 3.0.0 IndexReader to read the old version index and then use a
> 3.0.0 IndexWriter to write all the documents into a new index
> 3. Update QueryPaser to 3.0.0
>
> I've redeployed my system and it works fine now.
>
>
> On Wed, Dec 9, 2009 at 8:13 PM, Rob Staveley (Tom) <rstaveley@seseit.com
> >wrote:
>
> > I have Lucene 2.3.1 code and indexes deployed in production in a
> > distributed
> > system and would like to bring everything up to date with 3.0.0 via
> 2.9.1.
> >
> > Here's my migration plan:
> >
> > 1. Add a index writer which generates a 2.9.1 "test" index
> > 2. Have that "test" index writer push that 2.9.1 "test" index into
> > production and see if the distributed 2.3.1 index readers can cope with
> it
> > OK
> > 3. Upgrade my index writers to 2.9.1 (still using evil compressed fields)
> -
> > we shall have 2.9.1 writers and 2.3.1 readers during this phase. See if
> it
> > works.
> > 4. Upgrade my index readers to 2.9.1 (still using evil compressed fields,
> > but with support for explicit use of CompressionTools for decompression,
> > where fields have been explicitly compressed with CompressionTools - the
> > application will knows which need decompression)
> > 5. Add a CompressionTools to my "test" index writer, generating
> explicitly
> > compressed fields in the 2.9.1 "test" index
> > 6. Test explicit decompression for relevant fields with CompressionTools
> in
> > my 2.9.1 "test" index in my index readers
> > 7. Upgrade my "test" index writer to 3.0.0
> > 8. Have that "test" index writer push that 3.0.0 "test" index into
> > production and see if the distributed 2.9.1 index readers can cope with
> it
> > OK
> > 9. Go through my index writers and index reader clients and
> systematically
> > purge all of the Field.Store.COMPRESS fields and migrate to an explicit
> > CompressionTools approach where applicable and no compression where
> > applicable. During this phase I'll expect to have
> > CompressionTools-compressed fields coexisting with their
> > Field.Store.COMPRESS predecessors, where index reader client use of
> > Field.Store.COMPRESS is in transit to the explicit decompression
> approach.
> > 10. Upgrade my index writers to 3.0.0
> > 11. Upgrade my index readers to 3.0.0
> >
> > I've simplified this a bit, because I shan't really be testing straight
> off
> > in production(!) - I'll test the migration plan in a test cluster first;
> > but
> > this gives you the idea about the path.
> >
> > I wanted to know if I should expect problems with this plan. I'm
> depending
> > on newer writers generating indexes for older readers and 3 is a major
> > number upgrade. It looks like I can get away with this in version 3, but
> > that's by no means a guarantee according to
> > http://wiki.apache.org/lucene-java/BackwardsCompatibility#File_Formats
> >
> > Does this sound like a good plan?
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> --
> Weiwei Wang
> Alex Wang
> 王巍巍
> Room 403, Mengmin Wei Building
> Computer Science Department
> Gulou Campus of Nanjing University
> Nanjing, P.R.China, 210093
>
> Homepage: http://cs.nju.edu.cn/rl/weiweiwang
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Weiwei Wang
Alex Wang
王巍巍
Room 403, Mengmin Wei Building
Computer Science Department
Gulou Campus of Nanjing University
Nanjing, P.R.China, 210093

Homepage: http://cs.nju.edu.cn/rl/weiweiwang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message