lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason rutherglen <jasonhus...@yahoo.com>
Subject Re: GData Server - Lucene storage
Date Sat, 03 Jun 2006 00:29:09 GMT
Is it possible to turn off directory locking with BDB?  How is the performance compared to
regular FSDirectory for queries?

----- Original Message ----
From: Andi Vajda <vajda@osafoundation.org>
To: java-dev@lucene.apache.org; jason rutherglen <jasonhusong@yahoo.com>
Sent: Friday, June 2, 2006 10:52:27 AM
Subject: Re: GData Server - Lucene storage


On Fri, 2 Jun 2006, jason rutherglen wrote:

> It might be interesting to merge using BDB into Solr, as an option to 
> provide better realtime updates.  Perhaps the replication could be used as 
> well in place of rsync?  I don't have any experience with BDB replication, 
> anyone have thoughts on the matter?

If you're thinking of using Berkeley DB as a the store behind the Lucene index 
via the DbDirectory Directory implementation, here are a few things to keep in 
mind:

   - always setUseCompoundFile(false)
     don't use compound lucene index files on top of Berkeley DB:
      . there is a bug that prevents this from working correctly
      . it makes no sense anyway since it duplicates what DbDirectory is
        already doing (all index files are stored in the same Berkeley DB file)
      . it slows things down

   - if you are using a transaction around all the index updates, you may want
     to consider doing all the index updates in a RAMDirectory first and then
     adding the RAMDirectory wholesale to the DbDirectory in that transaction.
     This makes indexing considerably faster (3 times for me) and does a LOT
     less thrashing around in Berkeley DB which can lead to a large number of
     transactional log files rapidly filling up your hard drive.

I'm not really sure if and how index merging works. For my use, having no 
merging is good enough since I never update existing documents, but always 
instead add a new version of them. The concept of version is tied to my 
application and each transaction corresponds to a new version.

Andi..





Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message