Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of uwe@thetaphi.de designates
 188.138.97.18 as permitted sender)
From: "Uwe Schindler" <uwe@thetaphi.de>
To: <java-user@lucene.apache.org>,
	<paul_t100@fastmail.fm>
References: <4EEB77AD.1070102@fastmail.fm>
In-Reply-To: <4EEB77AD.1070102@fastmail.fm>
Subject: RE: Why is the old value still in the index
Date: Fri, 16 Dec 2011 18:43:07 +0100
Message-ID: <00bc01ccbc1a$2d9fa2f0$88dee8d0$@thetaphi.de>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-index: AQLsHqHtxs55zq33151pEab7v2gMUpOf9amg
Content-language: de

Hi,
> I'm adding documents to an index, at a later date I modify a document and
> update the index, close the writer and open a new IndexReader. My
> indexreader iterates over terms for that field and docFreq() returns one
as I
> would expect, however the iterator  returns both the old value of the
document
> and the new value, I don't expect (or want) the old value to still be in
the index,
> so why is this.

That is all as expected. Updating documents in a Lucene index is an atomic
delete/add operation. Deleting in Lucene just marks the document for
deletion, but it is still there (search results won't return it). The
condequence is that all terms are still in terms index and all document
frequencies still contain both documents. This *may* cause scoring problems
in indexes with many deletes (but those will go away as merging will remove
them, see below), but this is known (see wiki, javadocs,...).

Once you add more documents the index will merge segments and that will make
the deleted documents disappear. If you really want to do remove the old
documents with all terms (this is  veeeeery expensive), you can call
IW.forceMergeDeletes:
http://lucene.apache.org/java/3_5_0/api/core/org/apache/lucene/index/IndexWr
iter.html#forceMergeDeletes()

The way how inverted indexes work makes it impossible to update the terms
index afterwards.

Uwe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org