lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samarendra Pratap <samarz...@gmail.com>
Subject Re: Rewriting an index without losing 'hidden' data
Date Tue, 17 May 2011 12:51:59 GMT
Hi, I know it is too late to answer a question (sorry Chris) but I thought
it could be useful to share things (even late).
I was just going through the mails and I found that we've done it a few
months back.

*Objective: To add a new field to existing index without re-writing the
whole index.*

We have an index ("primary index") to which we want to add a new field,  say
"tags".
Source of the data is database.

I am adding pseudo code here

Create an index "index 2" with just two fields "id" (which is also a unique
identifier in main index) and "tags" (keep it stored) from database (source
of data).

Open a new IndexWriter ("index 3")

Now run a loop over all the documents of "Primary Index" with increasing
order of doc-id
Get document of current doc-id (starting from zero)
 Find the value of "id" field
Search this value in in secondary index in the same ("id") field.  (or
directly get the document through IndexReader and termVector). You should
get only one document.
 If document is found
Add this document to "index 3"
If document is not found
 Add a blank document to "index 3" (to maintain the doc-id order)

(After the loop is finished, the doc-ids and fields of "primary index" and
"index 3" will be in order, i.e. document at doc id 5 in "index 3" and in
"primary index" would be representing the same document of the database with
different fields)

Open a *ParallelReader* ( this is the key :-) ) and add both the indexes
("primary index" and "index 3") one by one.
Open an IndexWriter and use addIndexes(IndexReader) to create a single
index.
The final index will contain primary index with "tags" field. :-)


I request the list to comment if there could be any issue with that.


My question follows then -
I tried this on NumericField (as "tags") but this didn't work.
My guess (excuse me for guessing without deeper investigations) is that this
is because NumericField is not a Field. It is an AbstractField

Irrespective of the correctness of my guess can someone give me a hint or
point me to something which can help me doing the same process successfully
for NumericField as well?

I hope to listen from learned people.


On Fri, Apr 8, 2011 at 9:38 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> Unfortunately, updateDocument replaces the *entire* previous document
> with the new one.
>
> The ability to update a single indexed field (either replace that
> field entirely, or, change only certain token occurrences within it),
> while leaving all other indexed fields in the document unaffected, has
> been a long requested big missing feature in Lucene.  We call it
> "incremental field updates".
>
> There have been some healthy discussions on the dev list, that have
> worked out a good rough design (eg see
> http://markmail.org/thread/lsfjhpiblzymkfcn).  Also, recent
> improvements in how buffered deletes are handled should make it alot
> easier for updates to "piggyback" using that same packet stream
> approach.  So... I think there is hope some day that we'll get this
> into Lucene.
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Fri, Apr 8, 2011 at 11:00 AM, Ian Lea <ian.lea@gmail.com> wrote:
> > Unfortunately you just can't do this.  Might be possible if all fields
> > were stored but evidently they are not in your index.  For unstored
> > fields, the Document object will not contain the data that was passed
> > in when the doc was originally added.
> >
> > I believe there might be a way of recreating some of the missing data
> > via TermFreqVector but that has always sounded dodgy and lossy to me.
> >
> > The safest way is to reindex, however painful it might be.  Maybe you
> > could take the opportunity to upgrade lucene at the same time!
> >
> >
> > --
> > Ian.
> >
> >
> > On Fri, Apr 8, 2011 at 3:44 PM, Chris Bamford
> > <chris.bamford@talktalk.net> wrote:
> >> Hi,
> >>
> >> I recently discovered that I need to add a single field to every
> document in an existing (very large) index.  Reindexing from scratch is not
> an option I want to consider right now, so I wrote a utility to add the
> field by rewriting the index - but this seemed to lose some of the fields
> (indexed, but not stored?).  In fact, it shrunk a 12Gb index down to 4.2Gb -
> clearly not what I wanted.  :-)
> >> What am I doing wrong?
> >>
> >> My technique was:
> >>
> >>  Analyzer analyser = new StandardAnalyzer();
> >>  IndexSearcher searcher = new IndexSearcher(indexPath);
> >>  IndexWriter indexWriter = new IndexWriter(indexPath, analyser);
> >>  Hits hits = matchAllDocumentsFromIndex(searcher);
> >>
> >>  for (int i=0; i < hits.length(); i++) {
> >>          Document doc = hits.doc(i);
> >>          String id = doc.get("unique-id");
> >>          doc.add(new Field("newField", newValue, Field.Store.YES,
> Field.Index.UN_TOKENIZED));
> >>          indexWriter.updateDocument(new Term("unique-id", id), doc);
> >>  }
> >>
> >>  searcher.close();
> >>  indexWriter.optimize();
> >>  indexWriter.close();
> >>
> >> Note that my matchAllDocumentsFromIndex() does get the right number of
> hits from the index - i.e. the same number as held in the index.
> >>
> >>
> >>  Thanks for any ideas!
> >> BTW I am using Lucene 2.3.2.
> >>
> >> - Chris
> >>
> >>
> >>
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Regards,
Samar

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message