lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: IndexWriter.flush dose not flush the segmentInfos in Lucene 2.3
Date Tue, 25 Mar 2008 09:47:09 GMT

Ahhh, OK.  In 2.3, when you open IndexWriter with autoCommit=false,  
the only way to make changes visible is to close the writer.  Can you  
just open with autoCommit=true?  Then flush() makes changes visible.   
You can increase the flushing criteria to absurdly large values so  
the writer never flushes on its own, if that's necessary.

In 2.4, even when autoCommit=false, you can make changes visible  
(and, durable to power outage or OS/machine crash) by calling commit 
().  But remember 2.4 is trunk so "use at your own risk"!

Both flush() and commit() are rather costly operations, so calling  
them too frequently will hurt your indexing throughput.

Mike

Liu_Chao@emc.com wrote:
> I used the new released 2.3 and set autoCommit=false.
>
> Sorry my code to create the IndexWriter is like this:
> 	      Directory dir = FSDirectory.getDirectory(INDEX_DIR);
> 	      IndexWriter writer = new IndexWriter(dir, false, new  
> StandardAnalyzer(), true);
> 	      writer.setTermIndexInterval(8);
> 	      writer.setMaxBufferedDocs(3);
> 	      writer.setMergeFactor(3);
>
> I just would like to control the flush of the segments by myself.
>
> But in the released 2.3, there is no interface for users to flush  
> the segments by themselves, right?
>
> If I would like to achieve this, do you mean I have to use the 2.4  
> version instead of 2.3?
>
> Thank you
>
> Yours
>
> Liu Chao
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: 2008年3月25日 17:21
> To: java-dev@lucene.apache.org
> Subject: Re: IndexWriter.flush dose not flush the segmentInfos in  
> Lucene 2.3
>
>
> Do you mean 2.4 (trunk) not 2.3?  In 2.4, flush has been deprecated.
> Instead, you should use commit() to force the changes to be visible
> to a reader.
>
> On 2.3 I believe this works correctly as long as the write is opened
> with autoCommit=true (which your example below is).
>
> Mike
>
> Liu_Chao@emc.com wrote:
>> I found that for lucene 2.3, IndexWriter.flush does not flush the
>> segmentInfos.
>>
>> For example, I create a new index directory and add 10 documents to
>> it:
>>
>> 	IndexWriter writer = new IndexWriter(dir, new StandardAnalyzer(),
>> true);
>> 	indexDocs(writer, SOURCE_DIR);//in the source directory, there are
>> 10 documents.
>> 	writer.flush();
>>
>> If the IndexWriter is not closed and then an IndexReader is opened
>> to this index:
>>
>> 	IndexReader reader = IndexReader.open(INDEX_DIR);
>> 	int docNum = reader.numDocs();//docNum = 0
>>
>> at this time, docNum = 0, the new opened IndexReader cannot see the
>> new added documents.
>>
>> But if the IndexWriter is closed and then an IndexReader is opened
>> to the Index:
>>
>> 	writer.close();
>> 	IndexReader reader = IndexReader.open(index);
>> 	int docNum = reader.numDocs();//docNum = 10
>>
>> at this time the new opened IndexReader can see the documents added
>> by the IndexWriter.
>>
>> I think the reason is :
>>
>> IndexWriter.flush only flushes the data from memory to disk, the
>> metadata of segments segmentInfos is not flushed to the disk.
>>
>> So that if the IndexWriter is not closed, the new opened
>> IndexReader cannot see the new added documents.
>>
>> That means what the IndexReader can see is not the segment
>> structure when it is opened but when the last time the IndexWriter
>> is opened.
>>
>>
>> I think it is a bug for new version of lucene 2.3 or at least the
>> behavior is different from lucene 2.1
>>
>>
>> Yours
>>
>> Liu Chao
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message