lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Della Bitta <michael.della.bi...@appinions.com>
Subject Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Date Tue, 26 Mar 2013 20:02:13 GMT
Nate,

We just cleared up a problem similar to this by ditching Elastic Load
Balancer and switching over to the APR connector in Tomcat. Are you
using either of those?

Michael Della Bitta

------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic
<otis.gospodnetic@gmail.com> wrote:
> Hi Nate,
>
> Try adding some warmup queries and making sure the setting for using
> the cold searcher in solrconfig.xml is set to false.  Your warmup
> queries should use facets and sorting if your normal queries use them.
>  In SPM you'll actually see how much time warming up takes, so you'll
> get a better idea of the "cost" of that (when you don't do it).
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <nfox@neogov.com> wrote:
>> I was wondering if the warmup stuff was one of the culprits (we dont have
>> warmup's at all - the configs are pretty stock).
>> As for the system, it seems capable of quite a bit more: memory usage is
>> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and
>> load below 1.00.
>>
>> The seed data and queries were put together by one of our developers. I've
>> put all the solrmeter files here:
>> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
>> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure
>> which file does which specifically.
>>
>> Does the system's reaction to a 'fast load' without a warmup sound normal?
>> I would have expected the first couple hundred queries to be very slow
>> (>500ms) and then the system catch up after a while. But it just dies very
>> quickly and never recovers.
>>
>> I'll check out your SPM - I've seen it mentioned before. Thanks!
>>
>>
>>
>> --
>> Nate Fox
>> Sr Systems Engineer
>>
>> o: 310.658.5775
>> m: 714.248.5350
>>
>> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
>> Facebook<http://www.facebook.com/neogov>
>>
>> NEOGOV <http://www.neogov.com/> is among the top fastest growing software
>> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
>> the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
>>
>>
>>
>> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> In short, certain data structures need to load from index in the
>>> beginning, (for sorting and faceting) caches need to warm up, JVM
>>> needs to warm up, etc., so going slowly in the beginning makes sense.
>>> Why things die after that is a different Q.  Maybe it OOMs?  Maybe
>>> queries are very complex?  What do your queries look like?  I see
>>> newrelic.jar in the command-line.  May want to try SPM for Solr, it
>>> has better Solr metrics.
>>>
>>> Otis
>>> --
>>> Solr & ElasticSearch Support
>>> http://sematext.com/
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nfox@neogov.com> wrote:
>>> > I'm new to solr and I'm load testing our setup to see what we can handle.
>>> > I'm using solrmeter and my problem is a bit odd:
>>> > * When I set solrmeter to run 4000 queries/min, it will handle a few
>>> > hundred queries and then tomcat will stop responding completely to
>>> requests
>>> > (even though according to lsof -i it is still listening and the java
>>> > process is still running).
>>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can stop
>>> > solrmeter after a couple of  minutes at that pace and then run at
>>> 4000/min
>>> > without issue.
>>> >
>>> > It's as if it needs a ramp up time? Also, I noticed (regardless of ramp
>>> up)
>>> > that my setup cannot handle 8000/min. The reaction at 8k/min is the same
>>> as
>>> > if I were to run 4k/min without the ramp up. Of note, only the shard that
>>> > solrmeter is pointed to stops responding. The other shard hums along
>>> > without incident.
>>> >
>>> > Setup (everything in AWS):
>>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
>>> > (open-jdk-7-headless) : Ubuntu 12.04
>>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
>>> > I have ~30k documents in each node (~300Mb on each node)
>>> >
>>> > The vast majority of my solr/tomcat7 config is default from ubuntu's
>>> > packages/solr's example dir. Here's the configs and the end of the
>>> > catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
>>> >
>>> > My main question is two fold:
>>> > 1. Is this normal behavior for tomcat (to just stop responding
>>> completely)
>>> > when it gets overwhelmed? And the only option is to restart it? I guess
I
>>> > dont know what it looks like when tomcat/solr cant keep up.
>>> > 2. Why does it handle better when I give it a lower number of queries and
>>> > then ramp it up? It concerns me that if I have to restart a server in the
>>> > cluster and it gets thrown into the pool of machines that things will
>>> blow
>>> > up.
>>> >
>>> > As an aside, does this seem like a normal amount of queries (~4k/min)
>>> that
>>> > this kind of environment should be able to handle?
>>>

Mime
View raw message