Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <46B2021D.7070404@hannonhill.com>
Date: Thu, 02 Aug 2007 12:11:09 -0400
From: Zach Bailey <zach.bailey@hannonhill.com>
Organization: Hannon Hill
User-Agent: Thunderbird 2.0.0.6 (Windows/20070728)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Clustered Indexing on common network filesystem
References: <46AF61DD.4050704@hannonhill.com>
 <46B1EF1D.4040900@hannonhill.com> <11966423.post@talk.nabble.com>
 <46B1F952.7080403@hannonhill.com> <46B1FC03.6090607@gmail.com>
In-Reply-To: <46B1FC03.6090607@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Mark,

Thanks so much for your response.

Unfortunately, I am not sure the leader of the project would feel good 
about running code from trunk, save without an explicit endorsement from 
a majority of the developers or contributors for that particular code 
(do those people keep up with this list, anyway?) Is there any word on 
the possible timeframe the code required to work with NFS might be released?

Thanks for your other insight about hardlinks and rsync. I will look 
into that; unfortunately it does not cover our userbase who may be 
clustering in a Windows Server environment. I still have not heard/seen 
any evidence (anecdotal or otherwise) about how well lucene might work 
sharing indexes over a mounted Windows share.

-Zach

Mark Miller wrote:
> Some quick info:
> 
> NFS should work, but I think youll want to be working off the trunk. 
> Also, Sharing an index over NFS is supposed to be slow. The standard so 
> far, if you are not partitioning the index, is to use a unix/linux 
> filesystem and hardlinks + rsync to efficiently share index changes 
> across nodes (hard links for instant copy, rsync to only transfer 
> changed index files, search the mailing list). If you look at solr you 
> can see scripts that give an example of this. I don't think the scripts 
> rely on solr. This kind of setup should be quick and simple to 
> implement. Same with NFS. An RMI solution that allowed for index 
> partitioning would probably be the longest to do.
> 
> -Mark
> 
> 
> 
> Zach Bailey wrote:
>> Thanks for your response --
>>
>> Based on my understanding, hadoop and nutch are essentially the same 
>> thing, with nutch being derived from hadoop, and are primarily 
>> intended to be standalone applications.
>>
>> We are not looking for a standalone application, rather we must use a 
>> framework to implement search inside our current content management 
>> application. Currently the application search functionality is 
>> designed and built around Lucene, so migrating frameworks at this 
>> point is not feasible.
>>
>> We are currently re-working our back-end to support clustering (in 
>> tomcat) and we are looking for information on the migration of Lucene 
>> from a single node filesystem index (which is what we use now and hope 
>> to continue to use for clients with a single-node deployment) to a 
>> shared filesystem index on a mounted network share.
>>
>> We prefer to use this strategy because it means we do not have to have 
>> two disparate methods of managing indexes for clients who run in a 
>> single-node, non-clustered environment versus clients who run in a 
>> multiple-node, clustered environment.
>>
>> So, hopefully here are some easy questions someone could shed some 
>> light on:
>>
>> Is this not a recommended method of managing indexes across multiple 
>> nodes?
>>
>> At this point would people recommend storing an individual index on 
>> each node and propagating index updates via a JMS framework rather 
>> than attempting to handle it transparently with a single shared index?
>>
>> Is the Lucene index code so intimately tied to filesystem semantics 
>> that using a shared/networked file system is infeasible at this point 
>> in time?
>>
>> What would be the quickest time-to-implementation of these strategies 
>> (JMS vs. shared FS)? The most robust/least error-prone?
>>
>> I really appreciate any insight or response anyone can provide, even 
>> if it is a short answer to any of the related topics, "i.e. we 
>> implemented clustered search using per-node indexing with JMS update 
>> propagation and it works great", or even something as simple as "don't 
>> use a shared filesystem at this point".
>>
>> Cheers,
>> -Zach
>>
>> testn wrote:
>>> Why don't you check out Hadoop and Nutch? It should provide what you are
>>> looking for.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org