lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shalin Shekhar Mangar <shalinman...@gmail.com>
Subject Re: 2 strange behaviours with DIH full-import.
Date Sun, 01 Mar 2009 19:33:19 GMT
On Tue, Feb 17, 2009 at 5:31 PM, Marc Sturlese <marc.sturlese@gmail.com>wrote:

>
> 2.)I run a full-import and everythins works fine... I run another
> full-import in the same core and everything seems so work find. But I have
> noticed that the index in  /data/index dir is two times bigger. I have seen
> that Solr uses this indexwriter constructor when executes a deleteAll at
> the
> begining of the full import :
>
> http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/index/IndexWriter.html#IndexWriter(org.apache.lucene.store.Directory,%20org.apache.lucene.analysis.Analyzer,%20boolean,%20org.apache.lucene.index.IndexDeletionPolicy,%20org.apache.lucene.index.IndexWriter.MaxFieldLength)<http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/index/IndexWriter.html#IndexWriter%28org.apache.lucene.store.Directory,%20org.apache.lucene.analysis.Analyzer,%20boolean,%20org.apache.lucene.index.IndexDeletionPolicy,%20org.apache.lucene.index.IndexWriter.MaxFieldLength%29>
>
> Why lucene is not deleteing the data of the old index if the boolean var of
> the constructor is set to true? (the results are not duplicated but
> phisically the directory /index is double size). Has this something to do
> with de deletionPolicy that is saving commits or a lucenes 2.9-dev bug or
> something like that???
>

I think this is due to the IndexDeletionPolicy. The problem is that on a
commit, the IndexWriter is closed. It is re-opened only when you send
another add/delete command. If the index writer is closed, the deletion
policy does not take affect and unused commit points are not marked for
deletion. Replication also hit a similar problem, where the index files on
the slave were not getting cleaned up. The solution is the same, we need to
re-open the index writer after the commit closes it.

I'll open an issue and attach a fix.

-- 
Regards,
Shalin Shekhar Mangar.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message