lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Vasilev <>
Subject Re: Updating tag-indexes
Date Tue, 19 Aug 2008 15:33:03 GMT
Thanks Michael and Erick,

Yes we have our own unique IDs for the docs - we use them for other 
purposes, but here we need to deal with Lucene IDs as it is important 
for ParalelReader.
Ok then I will implement the algorithm for updating the small tag-index 
as I intended and as it is not guarantied from the API I will take care 
to check if docIDs remain to be assigned sequentially in each new Lucene 
I will also take care about deletions - first before starting update 
process for the small index I will keep info about deleted docs, then 
will undelete them. After that will start update algorithm. When it 
finishes if there are deletions this will mean that some exception 
occurred. The same will mean if the number of the docs is less than 
initial (then some collapse of deletions occurred) - will have to 
restart the process. If everything is ok then will delete again the docs 
about I have kept info in the begining.

Thanks Once Again,

Michael McCandless wrote:
> Yes, docIDs are currently sequentially assigned, starting with 0.
> BUT: on hitting an exception (say in your analyzer) it will usually 
> use up a docID (and then immediately mark it as deleted).
> Also, this behavior isn't "promised" in the API, ie it could in theory 
> (though I think it unlikely) change in a future release of Lucene.
> And remember when a merge completes (or, optimize), any deleted docs 
> will "collapse down" all docIDs after them.
> Mike
> Ivan Vasilev wrote:
>> Hi Lucene Guys,
>> I have a question that is simple but is important for me. I did not 
>> found the answer in the javadoc so I am asking here.
>> When adding Document-s by the method IndexWriter.addDocument(doc) 
>> does the documents obtain Lucene IDs in the order that they are added 
>> to the IndexWriter? I mean will first added doc be with Lucene ID 0, 
>> second added with Lucene ID 1, etc?
>> Bellow I describe why I am asking this.
>> We plan to split our index to two separate indexes that will be read 
>> by ParallelReader class. This is so because the one of them will 
>> contain field(s) that will be indexed and stored and it will be 
>> frequently changed. So to have always correct data returned from the 
>> ParallelReader when changing documents in the small index the Lucene 
>> IDs of these docs have to remain the same.
>> To do this Karl Wettin suggests a solution described in *LUCENE-879 
>> <>*. I do not like 
>> this solution because it is connected to changing Lucene source code, 
>> and after each refactoring potentially I will have problems. The 
>> solution is related to optimizing index so it will not be reasonably 
>> faster than the one that I prefer. And it is:
>> 1. Read the whole index and reconstruct the documents including index 
>> data by using TermDocs and TermEnum classes;
>> 2. Change the needed documents;
>> 3. Index documents in new index that will replace the initial one.
>> I can even simplify this algorithm (and the speed) if all the fields 
>> will be always stored - I can read just the stored data and based on 
>> this to reconstruct the content of the docs and re index them in new.
>> But anyway everything in the my approaches will depend on this - are 
>> LuceneIDs in the index ordered in the same way as docs are added to 
>> the IndexWriter.
>> Thanks in Advance,
>> Ivan
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:
> __________ NOD32 3366 (20080819) Information __________
> This message was checked by NOD32 antivirus system.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message