lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <>
Subject Re: Numerous problems with SolrCloud
Date Mon, 21 Dec 2015 16:21:05 GMT
ZK isn't pushed all that heavily, although all things are possible. Still,
for maintenance putting Zk on separate machines is a good idea. They
don't have to be very beefy machines.

Look in your logs for LeaderInitiatedRecovery messages. If you find them
then _probably_ you have some issues with timeouts, often due to
excessive GC pauses, turning on GC logging can help you get
a handle on that.

Another "popular" reason for nodes going into recovery is Out Of Memory
errors, which is easy to do in a system that gets set up and
then more and more docs get added to it. You either have to move
some collections to other Solr instances, get more memory to the JVM
(but watch out for GC pauses and starving the OS's memory) etc.

But the Solr logs are the place I'd look first for any help in understanding
the root cause of nodes going into recovery.


On Mon, Dec 21, 2015 at 8:04 AM, John Smith <> wrote:
> Thanks, I'll have a try. Can the load on the Solr servers impair the zk
> response time in the current situation, which would cause the desync? Is
> this the reason for the change?
> John.
> On 21/12/15 16:45, Erik Hatcher wrote:
>> John - the first recommendation that pops out is to run (only) 3 zookeepers, entirely
separate from Solr servers, and then as many Solr servers from there that you need to scale
indexing and querying to your needs.  Sounds like 3 ZKs + 2 Solr’s is a good start, given
you have 5 servers at your disposal.
>> —
>> Erik Hatcher, Senior Solutions Architect
>> <>
>>> On Dec 21, 2015, at 10:37 AM, John Smith <> wrote:
>>> This is my first experience with SolrCloud, so please bear with me.
>>> I've inherited a setup with 5 servers, 2 of which are Zookeeper only and
>>> the 3 others SolrCloud + Zookeeper. Versions are respectively 5.4.0 &
>>> 3.4.7. There's around 80 Gb of index, some collections are rather big
>>> (20Gb) and some very small. All of them have only one shard. The bigger
>>> ones are almost constantly being updated (and of course queried at the
>>> same time).
>>> I've had a huge number of errors, many different ones. At some point the
>>> system seemed rather stable, but I've tried to add a few new collections
>>> and things went wrong again. The usual symptom is that some cores stop
>>> synchronizing; sometimes an entire server is shown as "gone" (although
>>> it's still alive and well). When I add a core on a server, another (or
>>> several others) often goes down on that server. Even when the system is
>>> rather stable some cores are shown as recovering. When restarting a
>>> server it takes a very long time (30 min at least) to fully recover.
>>> Some of the many errors I've got (I've skipped the warnings):
>>> - org.apache.solr.common.SolrException: Error trying to proxy request
>>> for url
>>> - org.apache.solr.update.processor.DistributedUpdateProcessor; Setting
>>> up to try to start recovery on replica
>>> - org.apache.solr.common.SolrException; Error while trying to recover.
>>> core=[...]:org.apache.solr.common.SolrException: No registered leader
>>> was found after waiting
>>> - update log not in ACTIVE or REPLAY state. FSUpdateLog{state=BUFFERING,
>>> tlog=null}
>>> -; Could not publish as ACTIVE
>>> after succesful recovery
>>> - org.apache.solr.common.SolrException; Could not find core to call recovery
>>> - org.apache.solr.common.SolrException: Error CREATEing SolrCore '...':
>>> Unable to create core
>>> - org.apache.solr.request.SolrRequestInfo; prev == info : false
>>> - org.apache.solr.request.SolrRequestInfo; Previous SolrRequestInfo was
>>> not closed!
>>> - org.apache.solr.update.SolrIndexWriter; Error closing IndexWriter
>>> - org.apache.solr.update.SolrIndexWriter; SolrIndexWriter was not closed
>>> prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
>>> -; Error from shard
>>> - org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting
>>> for connection from pool
>>> - and so on...
>>> Any advice on where I should start? I've checked disk space, memory
>>> usage, max number of open files, everything seems fine there. My guess
>>> is that the configuration is rather unaltered from the defaults. I've
>>> extended timeouts in Zookeeper already.
>>> Thanks,
>>> John

View raw message