Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: Re: Jetty maxThreads
To: solr-user@lucene.apache.org
References: <250C760C-6E6C-41C2-A51C-BBFC098C9DDD@wunderwood.org>
From: Shawn Heisey <apache@elyograg.org>
Message-ID: <302caa7c-e6e8-d962-7e4e-70c58b471923@elyograg.org>
Date: Fri, 20 Oct 2017 15:46:18 -0600
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <250C760C-6E6C-41C2-A51C-BBFC098C9DDD@wunderwood.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
Content-Language: en-US
archived-at: Fri, 20 Oct 2017 21:46:33 -0000

On 10/18/2017 2:41 PM, Walter Underwood wrote:
> Jetty maxThreads is set to 10,000 which seams way too big.
>
> The comment suggests 5X the number of CPUs. We have 36 CPUs, which would mean 180 threads, which seems more reasonable.

I have not seen any evidence that maxThreads at 10000 causes memory
issues.� The out-of-the-box heap size for all recent releases is 512MB,
and Solr starts up just fine with 10000 maxThreads.

Most containers (including Jetty and Tomcat) default to a maxThreads
value of 200.� The Jetty included with Solr has had a setting of 10000
since I first started testing with version 1.4.� Users who provide their
own containers frequently run into problems where the container will not
allow Solr to start the threads it needs to run properly, so they must
increase the value.

This is a graph of threads on a couple of my Solr servers:

https://www.dropbox.com/s/4ux2y3xwvsjjrmt/solr-thread-graph.png?dl=0

The server named bigindy5 (rear graph in the screenshot) is my dev
server, running 6.6.2-SNAPSHOT.� The server named idxb6 is running
5.3.2-SNAPSHOT and is a backup production server.

The dev server has 8 CPU cores without hyperthreading and 24 Solr cores
(indexes).� Most of those cores have index data in them -- the dev
server has copies of *all* my indexes onboard.� It has very little
activity though -- aside from once-a-minute maintenance updates and a
monitoring server, there's virtually no query activity.

The production server has 20 CPU cores with hyperthreading (so it looks
like 40 to the OS), the same 24 Solr cores, but only a handful of those
cores have data, the rest are idle.� There's one critical difference in
activity for this server compared to the dev server -- four of the cores
on the machine are actively indexing from MySQL with the dataimport
handler, because I'm doing a full rebuild on that index.� Because this
server is in a backup role currently, its query load is similar to the
dev server -- almost nothing.

These servers handle distributed indexes, but they are NOT running in
cloud mode.� If there were active queries, then more threads would be
needed than currently are running.� If there were more Solr cores
(indexes), then more threads would be needed.

My installation is probably bigger than typical, but is definitely not
what I would call large.� As you can see from the screenshot, these
servers have reached thread counts in the 300 range, and are currently
sitting at about 250.� If I followed that recommendation of 5 threads
per CPU, I would configure a value of 40 on the dev server, which
wouldn't be anywhere near enough.

I've got another server running version 4.7.2 with 8 CPU cores (no
hyperthreading) and slightly fewer Solr cores.� This is a server that
actively gets queries at a fairly low QPS rate.� It shows a steady
thread count of around 200, with a peak thread count of 1032.� That
instance of Solr has an uptime of 208 days.

Based on what I have seen on my servers, I would not run with maxThreads
less than 2000, and I don't see any reason to change it from the
provided default of 10000.

Thanks,
Shawn