Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: Re: High Cpu sys usage
To: solr-user@lucene.apache.org
References: 
 <CAG1CohRtgQ3rXS-zyygM4K-nOL7TfDTFvnZqi3DVa1FSuz+oow@mail.gmail.com>
 <56DC4C1B.802@elyograg.org> <1457342634.4503.42.camel@tokemon>
From: Shawn Heisey <apache@elyograg.org>
Message-ID: <56DE3CCB.1030708@elyograg.org>
Date: Mon, 7 Mar 2016 19:45:31 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <1457342634.4503.42.camel@tokemon>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

On 3/7/2016 2:23 AM, Toke Eskildsen wrote:
> How does this relate to YouPeng reporting that the CPU usage increases?
>
> This is not a snark. YouPeng mentions kernel issues. It might very well
> be that IO is the real problem, but that it manifests in a non-intuitive
> way. Before memory-mapping it was easy: Just look at IO-Wait. Now I am
> not so sure. Can high kernel load (Sy% in *nix top) indicate that the IO
> system is struggling, even if IO-Wait is low?

It might turn out to be not directly related to memory, you're right
about that.  A very high query rate or particularly CPU-heavy queries or
analysis could cause high CPU usage even when memory is plentiful, but
in that situation I would expect high user percentage, not kernel.  I'm
not completely sure what might cause high kernel usage if iowait is low,
but no specific information was given about iowait.  I've seen iowait
percentages of 10% or less with problems clearly caused by iowait.

With the available information (especially seeing 700GB of index data),
I believe that the "not enough memory" scenario is more likely than
anything else.  If the OP replies and says they have plenty of memory,
then we can move on to the less common (IMHO) reasons for high CPU with
a large index.

If the OS is one that reports load average, I am curious what the 5
minute average is, and how many real (non-HT) CPU cores there are.

Thanks,
Shawn