lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: Lucene 2.3.0 and NFS
Date Fri, 04 Apr 2008 09:16:14 GMT

Rajesh parab wrote:
> Hi,
> We are currently using Lucene 2.0 for full-text
> searches within our enterprise application, which can
> be deployed in clustered environment. We generate
> Lucene index for data stored inside relational
> database.
> As Lucene 2.0 did not have solid NFS support and as we
> wanted Lucene based searches to work properly in
> Clustered environment, we had decided on following
> approach:
> 1. The index generation happens on a machine (could be
> one of the cluster nodes or a separate machine) and
> once the Lucene index is generated, we copy all the
> index files to the database.

Note that you can do also incremental replication: often, the Lucene  
index changes in minor ways (eg a single new segment is flushed and a  
new segments_N and segments.gen is written) so you should only sync  
the files that are new (and remove the ones that are now gone).   
Lucene's write-once approach makes this very simple (you just have to  
compare file names, not the contents of each file).

It's also possible to replicate without using a DB.  EG rsync does a  
great job.

> 2. The index search request on each cluster node
> retrieves the index files from database (during first
> search or after index update), copies to the file
> system and use it for searches.
> 3. Thus, each cluster node has its own copy of the
> index and it keeps on picking up latest version if it
> is available inside database.
> This has worked fine for us till now, though we will
> not be able continue with this model in future as we
> want to support Lucene based searches across our
> application and also want to index large components
> inside our application like Wiki, forums, etc. As the
> index will grow, storing and retrieving index files
> from database will not be an efficient operation.
> My questions are:
> - Will we be able to use NFS if we move to Lucene
> 2.3.0?

Make sure you update to 2.3.1, not 2.3.0.

> - Will there be any significant performance impact on
> index generation and searches if we move to NFS?
> - Is Lucene + NFS combination supported for all
> operating systems? (We support Windows, Solaris, AIX,
> HP-UX, Red Hat Linux)

NFS *should* work, however:

   * It's not widely used, so, test thoroughly in your particular setup.

   * Most likely to work is if you use a single machine writing to  
the index, and many readers.

   * Performance is likely not great, especially on searching, but  
you should test in your specific situation.

> - Is there any other alternative available other than
> NFS?

It's also possible to replicate without using a DB.  EG rsync does  
agreat job.

You should look at Solr, since it already has all the infrastructure  
toaccept updates, replicate index changes to remote machines, etc.

> I will really appreciate your comments/thoughts on
> this topic.
> Regards,
> Rajesh
> ______________________________________________________________________ 
> ______________
> You rock. That's why Blockbuster's offering you one month of  
> Blockbuster Total Access, No Cost.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message