lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Busch (JIRA)" <>
Subject [jira] Commented: (LUCENE-2312) Search on IndexWriter's RAM Buffer
Date Mon, 22 Mar 2010 17:41:27 GMT


Michael Busch commented on LUCENE-2312:

I think sync'ing after every doc is probably the better option.  We'll still avoid the need
to make all variables downstream of DocumentsWriter volatile/atomic, which should be a nice
performance gain.

The problem with the delayed sync'ing (after e.g. 100 docs) is that if you don't have a never-ending
stream of twee... err documents, then you might want to force an explicit sync at some point.
 But that's very hard, because you would have to force the writer thread to make e.g. a volatile
write via an API call.  And if that's an IndexWriter writer API that has to trigger the sync
on multiple DocumentsWriter instances (i.e. multiple writer threads) I don't see how that's
possible unless Lucene manages it's own thread of pools.

> Search on IndexWriter's RAM Buffer
> ----------------------------------
>                 Key: LUCENE-2312
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Search
>    Affects Versions: 3.0.1
>            Reporter: Jason Rutherglen
>            Assignee: Michael Busch
>             Fix For: 3.1
> In order to offer user's near realtime search, without incurring
> an indexing performance penalty, we can implement search on
> IndexWriter's RAM buffer. This is the buffer that is filled in
> RAM as documents are indexed. Currently the RAM buffer is
> flushed to the underlying directory (usually disk) before being
> made searchable. 
> Todays Lucene based NRT systems must incur the cost of merging
> segments, which can slow indexing. 
> Michael Busch has good suggestions regarding how to handle deletes using max doc ids.
> The area that isn't fully fleshed out is the terms dictionary,
> which needs to be sorted prior to queries executing. Currently
> IW implements a specialized hash table. Michael B has a
> suggestion here: 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message