lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Stewart <bstewart...@gmail.com>
Subject Re: Replication with an HA master
Date Thu, 13 Oct 2011 20:01:38 GMT
Yes that is a good point.  Thanks.

I think I will avoid using NAS/SAN and use two masters, one setup as a repeater (slave and
master).  In case of very rare master failure, some minor manual intervention will be required
to re-configure remaining master or bring other one back up.

My only concern in that case is losing new documents from solrj client since there is no broker/buffer/queue
between solrj client and SOLR master.  It would be nice if there was some open source broker/queue
which could sit between solrj and SOLR and queue up messages (publish/subscribe).

Bob

On Oct 13, 2011, at 3:56 PM, Jaeger, Jay - DOT wrote:

> One thing to consider is the case where the JVM is up, but the system is otherwise unavailable
(say, a NIC failure, firewall failure, load balancer failure) - especially if you use a SAN
(whose connection is different from the normal network).
> 
> In such a case the old master might have uncommitted updates.
> 
> JRJ
> 
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com] 
> Sent: Tuesday, October 11, 2011 3:17 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Replication with an HA master
> 
> Hello,
> ----- Original Message -----
> 
>> From: Robert Stewart <bstewart.ny@gmail.com>
>> To: solr-user@lucene.apache.org
>> Cc: 
>> Sent: Tuesday, October 11, 2011 3:37 PM
>> Subject: Re: Replication with an HA master
>> 
>> In the case of using a shared (SAN) index between 2 masters, what happens if the

>> live master fails in such a way that the index remains "locked" (such 
>> as if some hardware failure and it did not unlock/close index).  Will the other 
>> master be able to open/write to the index as new documents are added?
> 
> 
> You'd use native locks, which should disappear if the JVM dies.  If it does not, then
I'm not 100% sure what happens, but in the worst case there would be a need for a quick manual
(or scripted) intervention.  But your index would be up to date!
> 
>> Also, if that can work ok, would it work if you have a LB (VIP) from both 
>> indexing and replication sides of the 2 masters, such that some VIP used by 
>> solrj for indexing new documents via HTTP, and the same VIP used by slave 
>> searchers for replication?  That sounds like it would work.
> 
> 
> Precisely what you should do.  e.g. "master-vip" is the "hostname" that both SolrJ would
post new docs to and the master "server" slaves would poll for index changes.
> 
> Otis
> ----
> 
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
> 
> 
> 
> 
>> On Oct 11, 2011, at 3:16 PM, Otis Gospodnetic wrote:
>> 
>>> Hello,
>>> 
>>> Yes, you've read about NFS, which is why I gave the example of a SAN 
>> (which can have multiple power supplies, controllers, etc.)
>>> 
>>> Yes, should be OK to have multiple Solr instances have the same index open, 
>> since only one of them will actually be writing to it, thanks to LB.
>>> 
>>> Otis
>>> ----
>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>> Lucene ecosystem search :: http://search-lucene.com/
>>> 
>>> 
>>>> ________________________________
>>>> From: Brandon Ramirez <Brandon_Ramirez@elementk.com>
>>>> To: "solr-user@lucene.apache.org" 
>> <solr-user@lucene.apache.org>
>>>> Sent: Tuesday, October 11, 2011 2:55 PM
>>>> Subject: RE: Replication with an HA master
>>>> 
>>>> Using a shared volume crossed my mind too, but I discarded the idea 
>> because of literature I have read about Lucene performing poorly against remote 
>> file systems.  But then I suppose a SAN wouldn't be a remote file system in 
>> the same sense as an NFS-mounted NAS or similar.
>>>> 
>>>> Should I be concerned about two solr instances on two machines having 
>> the same SAN-based index open, as long as only one of them is receiving 
>> requests?  I would think in theory it would work, but I don't have any 
>> production-level experience with Solr yet, only textbook knowledge.
>>>> 
>>>> 
>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848 
>>>> Software Engineer II | Element K | www.elementk.com
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com] 
>>>> Sent: Tuesday, October 11, 2011 2:28 PM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: Re: Replication with an HA master
>>>> 
>>>> A few alternatives:
>>>> * Have the master keep the index on a shared disk (e.g. SAN)
>>>> * Use LB to easily switch to between masters, potentially even 
>> automatically if LB can detect the primary is down
>>>> 
>>>> Otis
>>>> ----
>>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene 
>> ecosystem search :: http://search-lucene.com/
>>>> 
>>>> 
>>>>> ________________________________
>>>>> From: Robert Stewart <bstewart.ny@gmail.com>
>>>>> To: solr-user@lucene.apache.org
>>>>> Sent: Friday, October 7, 2011 10:22 AM
>>>>> Subject: Re: Replication with an HA master
>>>>> 
>>>>> Your idea sounds like the correct path.  Setup 2 masters, one 
>> running 
>>>>> in "slave" mode which pulls replicas from the live 
>> master.  When/if live master goes down, you just reconfigure and restart the 
>> backup master to be the live master.  You'd also need to then start data 
>> import on the backup master (enable/start cron job?), and redirect slave 
>> searchers to pull replicas from the new live master.  All that could be done 
>> using scripts or something like puppet possibly.
>>>>> 
>>>>> Another solution maybe is to run 2 "live" masters, which 
>> both index the same content from the same data source.  If one goes down, then 
>> you just need to redirect slave searchers to the backup master for replication.
>>>>> 
>>>>> I am also starting a similar project which needs some disaster 
>> recovery processes in place, so any other info would be useful to me as well.
>>>>> 
>>>>> Bob
>>>>> 
>>>>> On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>>>>> 
>>>>>> We are getting ready to start a project using Solr as our 
>> backend search engine and I am trying to devise a deployment architecture that 
>> works for us.  We definitely need a master/slave replication strategy, 
>> that's for sure, but my concern is the master becomes a single point of 
>> failure.
>>>>>> 
>>>>>> Fortunately, real-time search is not a requirement for us.  If 
>> search results are a few minutes out of sync with our database, it's not a 
>> big deal.
>>>>>> 
>>>>>> So what I would like to do is have a set of query servers 
>> (slaves) that are only used for querying, no indexing and have them use 
>> Solr's HTTP replication mechanism on a 2 or 3 minute interval.  To get HA 
>> indexing, I'd like to have 2 masters: a primary and a standby.  All indexing 
>> requests go to the primary unless it's taken out of service.  To keep the 
>> standby ready to takeover if it needs to, it needs to be more up to date than 
>> the slaves.  I'd like to have it replicate every 30 seconds or so.
>>>>>> 
>>>>>> The reason I'm asking about it on this list is that I 
>> haven't seen any Solr documentation or even anything that talks about this.  
>> I can't be the only one concerned about having a single point of failure, so 
>> I'm reaching out to see what others have done in this case before I go with 
>> my own solution.
>>>>>> 
>>>>>> 
>>>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848 
>> Software 
>>>>>> Engineer II | Element K | 
>> www.elementk.com<http://www.elementk.com/>
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 


Mime
View raw message