Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Message-ID: <1446470220.3703.389.camel@te-prime>
Subject: Re: Very high memory and CPU utilization.
From: Toke Eskildsen <te@statsbiblioteket.dk>
Reply-To: <te@statsbiblioteket.dk>
To: <solr-user@lucene.apache.org>
Date: Mon, 2 Nov 2015 14:17:00 +0100
In-Reply-To: 
 <CAG3m1h4FV-7LHFMH0p6t2EETMeE2w=y5F+JPi8KvKEZXEX3z=Q@mail.gmail.com>
References: 
 <CAG3m1h49TX3uZkBCPhQQr1r5yagp1SKecJZViqBovC=3cTSV7A@mail.gmail.com>
	 <1446451247.3703.306.camel@te-prime>
	 <CAG3m1h7huYtfnFVBEwEa9YNX_+1xyOsj5h4ERB59yBqyq7Qrew@mail.gmail.com>
	 <CAG3m1h4_ay810S2m0=Z4idxcA78ctvS+v3wMExs0w4uJ=O1wTg@mail.gmail.com>
	 <CAKUpjcSuGfyyLVhEdZzN-LKUXFSMEocn9FZ2F1vS+QwXLczFGw@mail.gmail.com>
	 <CAKUpjcQVWtXW05tOWvX9=CMSxUJik9a2NOmHt1qqy+vHcaZOnw@mail.gmail.com>
	 <CAG3m1h5pD9G4XdqUou=qxCkRfvnVY0XK6Ez0g3dr_yywFpGNcg@mail.gmail.com>
	 <1446465034.3703.369.camel@te-prime>
	 <CAG3m1h4FV-7LHFMH0p6t2EETMeE2w=y5F+JPi8KvKEZXEX3z=Q@mail.gmail.com>
Organization: State and University Library, Denmark
Content-Type: text/plain; charset="UTF-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

On Mon, 2015-11-02 at 17:27 +0530, Modassar Ather wrote:

> The query q=network se* is quick enough in our system too. It takes
> around 3-4 seconds for around 8 million records.
> 
> The problem is with the same query as phrase. q="network se*".

I misunderstood your query then. I tried replicating it with
q="der se*"

http://rosalind:52300/solr/collection1/select?q=%22der+se*%
22&wt=json&indent=true&facet=false&group=true&group.field=domain

gets expanded to

parsedquery": "(+DisjunctionMaxQuery((content_text:\"kan svane\" |
author:kan svane* | text:\"kan svane\" | title:\"kan svane\" | url:kan
svane* | description:\"kan svane\")) ())/no_coord"

The result was 1,043,258,271 hits in 15,211 ms


Interestingly enough, a search for 
q="kan svane*"
resulted in 711 hits in 12,470 ms. Maybe because 'kan' alone matches 1
billion+ documents. On that note,
q=se*
resulted in -951812427 hits in 194,276 ms.

Now this is interesting. The negative number seems to be caused by
grouping, but I finally got the response time up in the minutes. Still
no memory problems though. Hits without grouping were 3,343,154,869.

For comparison,
q=http
resulted in -1527418054 hits in 87,464 ms. Without grouping the hit
count was 7,062,516,538. Twice the hits of 'se*' in half the time.

> I changed my SolrCloud setup from 12 shard to 8 shard and given each
> shard 30 GB of RAM on the same machine with same index size
> (re-indexed) but could not see the significant improvement for the
> query given.

Strange. I would have expected the extra free memory for disk space to
help performance.

> Also can you please share your experiences with respect to RAM, GC,
> solr cache setup etc as it seems by your comment that the SolrCloud
> environment you have is kind of similar to the one I work on?
> 
There is a short write up at
https://sbdevel.wordpress.com/net-archive-search/

- Toke Eskildsen, State and University Library, Denmark