Mailing-List: contact lucene-net-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucene-net-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of mmcconna@oxford-analytica.com
 designates 195.130.217.35 as permitted sender)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Date: Mon, 23 May 2011 17:19:16 +0100
Message-ID: 
 <71B5A2C57E0AD04F97E997AA244D455E01C90141@BEDIVERE.alfredst.oxford-analytica.local>
In-Reply-To: 
 <F9440C6FF96AEB4F996F317317322B920804E2B2@vDAYPEXCH1.CIN2.creditinfonet.com>
Thread-Topic: [Lucene.Net] Server farm sharing Lucene
Thread-Index: AcwXFPOzjZBHv0D+T/+EvO5lZ+8a8QAUcbOAAHf1gYAAAMjfgAAGm2iw
References: 
 <F9440C6FF96AEB4F996F317317322B920804CE8C@vDAYPEXCH1.CIN2.creditinfonet.com>
 <B39F47D9-4397-4311-B54B-4BE4589685FC@tpg.com.au>
 <71B5A2C57E0AD04F97E997AA244D455E01C90129@BEDIVERE.alfredst.oxford-analytica.local>
 <F9440C6FF96AEB4F996F317317322B920804E2B2@vDAYPEXCH1.CIN2.creditinfonet.com>
From: "Moray McConnachie" <mmcconna@oxford-analytica.com>
To: <lucene-net-user@lucene.apache.org>
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Subject: RE: [Lucene.Net] Server farm sharing Lucene

We use a message queue (MSMQ) for our indexing updates, and it works
well. It's also not a bad one-to-many distribution system so you can
update several search servers if you want to handle fail-over through
multiple updates (rather than synchronising indices), and it can handle
transactions, exceptions and retries pretty well too so you should be
able to guarantee updates get synchronised.=20

It's looked to me for a while as if MS will eventually get rid of MSMQ -
but I might be out-of-date.

I've had some bad experiences in the long distant past (v1.0) with
.NET's file monitoring with updates failing to fire. No doubt this is
all fixed in recent OS/.NET combinations, but it left me with a sour
taste.

M.

-------------------------------------
Moray McConnachie
Director of IT    +44 1865 261 600
Oxford Analytica  http://www.oxan.com

-----Original Message-----
From: Brian Sayatovic [mailto:bsayatovic@creditinfonet.com]=20
Sent: 23 May 2011 14:06
To: lucene-net-user@lucene.apache.org
Subject: RE: [Lucene.Net] Server farm sharing Lucene

Interesting!

Right now, the index updating occurs on the same thread where the DB
write is occurring.  This is nice in that we have little room for one to
happen without the other.  With a dedicated search server, I'd have to
see pushing the update off to that other server via a message queue,
perhaps, and then the ability have all servers in the farm query through
it.

Still, I'd worry about fail over.  We have some other failover
strategies where every server in the farm is capable of a function, but
only one server is actively doing it.  But each server periodically
checks if any other server still has an "active claim" (i.e. not too
old) and if not, it will pick up.  So in the event one server fails,
another in the farm takes over.

Perhaps I could marry the two.

But, as said earlier in this thread, I won't prematurely optimize.

-----Original Message-----
From: Moray McConnachie [mailto:mmcconna@oxford-analytica.com]
Sent: Monday, May 23, 2011 4:40 AM
To: lucene-net-user@lucene.apache.org
Subject: RE: [Lucene.Net] Server farm sharing Lucene

If your traffic is high enough to warrant the server farm, and search is
a highly used feature, it is also worth thinking about a dedicated
search server (or pair of such synced as suggested by Ken and/or
separately driven by your publishing tools depending on the degree of
redundancy and failsafe you need).

We use a dedicated search server as a service, running a custom wrapper
- we pass a Lucene Query across the network using .NET Remoting -
binary-serialization over TCP (stay away from other forms of
serialization unless you have lots of resources to throw at search and
lots of bandwidth), returning a custom object containing the results and
other assorted metadata, including faceting.

.NET remoting is a joy in this context, you only need to be careful
about version synchronisation - upgrades need to be carefully planned so
that servers with e.g. an upgraded Lucene only talk to a search server
with an upgraded Lucene.

Yours,
Moray
-------------------------------------
Moray McConnachie
Director of IT    +44 1865 261 600
Oxford Analytica  http://www.oxan.com

-----Original Message-----
From: Ken Foskey [mailto:kfoskey@tpg.com.au]
Sent: 21 May 2011 00:25
To: lucene-net-user@lucene.apache.org
Subject: Re: [Lucene.Net] Server farm sharing Lucene

Shared directory means network so you have two latencies and much more
traffic on the network.

.net has file monitor which will trigger a function on change of file.
You can use this to push a file on change.  If you do this copy it to
the same file system (partition) then move it into place after so it is
immediately copied.

Ken Foskey

On 21/05/2011, at 3:40 AM, Brian Sayatovic
<bsayatovic@creditinfonet.com> wrote:

> How have folks gone about setting up Lucene in a server farm?  Just a
network-accessible shared directory?
>
> Regards,
> Brian.
> ________________________________
>
> Learn more about the products, services and technology solutions=20
> available from CIN Legal Data Services at:
> www.cinlegal.com<http://www.cinlegal.com>
>
> This message may contain confidential / proprietary information from
CIN Legal Data Service and Credit Infonet, Inc.. If you are not an
intended recipient, please refrain from the disclosure, copying,
distribution or use of this information. All such unauthorized actions
are strictly prohibited. If you have received this transmission in
error, please notify the sender by e-mail at
bsayatovic@creditinfonet.com and delete all copies of this material from
any computer.

---------------------------------------------------------
Disclaimer

This message and any attachments are confidential and/or privileged. If
this has been sent to you in error, please do not use, retain or
disclose them, and contact the sender as soon as possible.

Oxford Analytica Ltd
Registered in England: No. 1196703
5 Alfred Street, Oxford
United Kingdom, OX1 4EH
---------------------------------------------------------

________________________________

Learn more about the products, services and technology solutions
available from CIN Legal Data Services at:
www.cinlegal.com<http://www.cinlegal.com>

This message may contain confidential / proprietary information from CIN
Legal Data Service and Credit Infonet, Inc.. If you are not an intended
recipient, please refrain from the disclosure, copying, distribution or
use of this information. All such unauthorized actions are strictly
prohibited. If you have received this transmission in error, please
notify the sender by e-mail at bsayatovic@creditinfonet.com and delete
all copies of this material from any computer.

---------------------------------------------------------
Disclaimer=20

This message and any attachments are confidential and/or privileged. If thi=
s has been sent to you in error, please do not use, retain or disclose them=
, and contact the sender as soon as possible.

Oxford Analytica Ltd
Registered in England: No. 1196703
5 Alfred Street, Oxford
United Kingdom, OX1 4EH
---------------------------------------------------------