lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Wartes <>
Subject Re: Solr performance on EC2 linux
Date Tue, 02 May 2017 04:18:55 GMT
I started with the same three-node 15-shard configuration I’d been used to, in an RF1 cluster.
(the index is almost 700G so this takes three r4.8xlarge’s if I want to be entirely memory-resident)
I eventually dropped down to a 1/3rd size index on a single node (so 5 shards, 100M docs each)
so I could test configurations more quickly. The system time usage was present on all solr
nodes regardless. I adjusted for a difference in the CPU count on the EC2 nodes when I picked
my load testing rates. 

Zookeeper is a separate cluster on separate nodes. It is NOT collocated with Solr, although
it’s dedicated exclusively to Solr’s use.

I specify a timeout on all queries, and as mentioned, use SOLR-4449. So there’s possibly
an argument I’m doing a lot more timing related calls than most. There’s nothing particularly
exotic there though, just another Executor Service, and you’ll never get a backup request
on an RF1 cluster because there’s no alternate to try. 

On 5/1/17, 6:28 PM, "Walter Underwood" <> wrote:

    Might want to measure the single CPU performance of your EC2 instance. The last time I
checked, my MacBook was twice as fast as the EC2 instance I was using.
    Walter Underwood,1,L0yDngRyy1MwN7dh5tRFW86sVcn6tcLZH4c03j0EdQSsGBMn0SLDqeB_sHQjB4DdbRMOLka5MnyeXnKS_CEUEv4qIgU5wuyhZBMHciVoH6e8uo7KGr09mXTtDw,,&typo=0
 (my blog)
    > On May 1, 2017, at 6:24 PM, Chris Hostetter <> wrote:
    > : tldr: Recently, I tried moving an existing solrcloud configuration from 
    > : a local datacenter to EC2. Performance was roughly 1/10th what I’d 
    > : expected, until I applied a bunch of linux tweaks.
    > How many total nodes in your cluster?  How many of them running ZooKeeper?
    > Did you observe the heavy increase in system time CPU usage on all nodes, 
    > or just the ones running zookeeper?
    > I ask because if your speculation is correct and it is an issue of 
    > clocksource, then perhaps ZK is where the majority of those system calls 
    > are happening, and perhaps that's why you didn't see any similar heavy 
    > system CPU load in ES?  
    > (Then again: at the lowest levels "lucene" really shouldn't care about 
    > anything clock related at all Any "time" realted code would live in the 
    > Solr level ... hmmm.)
    > -Hoss

View raw message