lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Clarification about segments
Date Tue, 26 Aug 2008 08:39:06 GMT

Before 2.3, each doc was in fact a separate segment in memory, and  
then these segments were merged together to flush a single segment in  
the Directory.

As of 2.3, IndexWriter now writes directly into RAM the data  
structures that are needed to create the segment, and then flushing  
the segment is a matter of copying these data structures into the  
Directory.  This gave a substantial speedup to indexing throughput,  
much better RAM efficiency (documents per MB that IndexWriter can  
buffer), etc.

In any event, for all versions of Lucene, when flush happens that  
flush adds a single new segment to the index.

Mike

David Lee wrote:

> ok, thanks. I knew that the documents were buffered in memory until  
> they
> were flushed, but I thought that in memory, they were still separate
> documents/segments until they were merged together at the  
> appropriate time
> (dependent on the mergeFactor).
>
> Do you mean that when the IndexWriter flushes the documents in  
> memory to the
> disk, it will merge all the documents in that flush to one segment?
>
> Thanks!
> David
>
> On Sat, Aug 23, 2008 at 2:40 AM, Karsten F.
> <karsten-lucene@fiz-technik.de>wrote:
>
>>
>> Hi David,
>>
>> this is not true, please take a look to
>> IndexWriter#setRAMBufferSizeMB
>> and
>> IndexWriter#setMaxBufferedDocs
>>
>> But you can produce 9 segments (each with only one document), if  
>> you call
>> IndexWriter#flush
>> or
>> IndexWriter#commit
>> after each addDocument
>>
>> so from my knowledge about lucene there is no difference between
>> #flush
>> and
>> #optimize(getMergeFactor())
>> (btw #optimize() is equal to optimize(1) ).
>>
>>
>> Best regards
>> Karsten
>>
>> p.s. and yes, searching goes through every segment.
>>
>>
>> David Lee-26 wrote:
>>>
>>> So from what I understand, is it true that if mergeFactor is 10,  
>>> then
>> when
>>> I
>>> index my first 9 documents, I have 9 separate segments, each  
>>> containing 1
>>> document? And when searching, it will search through every segment?
>>>
>>> Thanks!
>>> David
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Clarification-about-segments-tp19117115p19120086.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message