lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Marius Kirsch <>
Subject Augmenting an existing index (was: ACLs and Lucene)
Date Mon, 30 May 2005 20:20:52 GMT

I have a similar problem, for which ParallelReader looks like a good
solution -- except for the problem of creating a set of indices with
matching document numbers.

I want to augment the documents in an existing index with information
that can be extracted from the same index. (Basically, I am indexing a
mailing list archive, and want to add keyword fields to documents that
contain the message ids of followup messages. That way, I could
quickly link to the followup messages from original
message. Unfortunately, I don't know the ids of all followup messages
until after I indexed the whole archive.)

I tried to implement a FilterIndexReader that would add the required
information, but couldn't get that to work. (I guess there's more to
extending FilterIndexReader than just overriding the document() method
and tacking a few more keyword fields on to the document before
returning it.) When I add my FilterIndexReader to a new IndexWriter
with the .addIndexes() method, it seems to work, but when I try to
optimize the new index, I get the following error:

merging segments _0 (1900 docs)Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException:
	at java.util.ArrayList.get(
	at org.apache.lucene.index.FieldInfos.fieldInfo(
	at org.apache.lucene.index.FieldsReader.doc(
	at org.apache.lucene.index.SegmentReader.document(
	at org.apache.lucene.index.SegmentMerger.mergeFields(
	at org.apache.lucene.index.SegmentMerger.merge(
	at org.apache.lucene.index.IndexWriter.mergeSegments(
	at org.apache.lucene.index.IndexWriter.optimize(
	at org.sebastiankirsch.thesis.util.MailFilterIndexReader.main(

If I don't optimize the index, I don't get an error, but Luke cannot
read the new index properly. I guess this has something to do with me
messing with the documents without properly adjusting the index terms

At the moment, I index the whole archive twice, and use the info from
the first index to add missing fields to the second index. However, it
would save me a lot of work (and processing power, of course) if I
could just postprocess the index from the first pass without
re-indexing the messages. Furthermore, it would open up the
possibility to apply even more passes to the postprocessing. (I'm
probably going to need that soon.)

I presume that a ParallelIndexReader could be merged into a single
index using addIndexes()? So if the problem of keeping the doc numbers
in sync can be solved ...

Alternatively, I would welcome hints as to how to implement a
FilterIndexReader properly.

Thanks very much for your time, Sebastian

On Mon, May 30, 2005 at 11:32:13AM -0400, Robichaud, Jean-Philippe wrote:
> What about:
> ne/index/

Sebastian Kirsch <> []

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message