lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Bahr <r...@manzama.com>
Subject Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return
Date Wed, 04 Sep 2019 21:55:30 GMT
Hi Toke,
Please see below.  We reindexed the solr8 cluster to make sure it was up to
date with content.

* What is your Xmx for the Solrs? -
solr8

SOLR_JAVA_MEM="-Xms11235m -Xmx11235m" - 70% of OS Memory


solr4

export CATALINA_OPTS="-Xms11235m -Xmx11235m" - - 70% of OS Memory


(If you use most of the physical memory for Java, there might be too
little left for OS-caching)

* Do you have a large number of stored or docValued fields?
solr8
not sure if this would be considered a large amount or not.  Posted schema
in dropbox
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0
solr4
posted schema in dropbox
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0

(Solr 8 returns docValued fields per default while Solr 4 does not)

* How large is a response? One simple way to check is
curl '
http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
'
| wc -c

solr8

  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current

                                 Dload  Upload   Total   Spent    Left
Speed

100 18830  100 18830    0     0    245      0  0:01:16  0:01:16 --:--:--
5822

   18830

solr4

  % Total    % Received % Xferd  Average Speed   Time    Time     Time
Current

                                 Dload  Upload   Total   Spent    Left
Speed

100 13149    0 13149    0     0    807      0 --:--:--  0:00:16 --:--:--
3273

   13149



(Solr 4 and 8 compress stored content differently and a very large
result set requires more CPU power to uncompress in Solr 8 (but less
IO))

* Do you have any response related defaults in your solrconfig.xml,
such as faceting or grouping?

solr8
posted config
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0

solr4
posted config
https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0

(You might be doing heavy aggregation even if you don't explicitly ask
for it)


*Manzama*a MODERN GOVERNANCE company

Russell Bahr
Lead Infrastructure Engineer

USA & CAN Office: +1 (541) 306 3271
USA & CAN Support: +1 (541) 706 9393
UK Office & Support: +44 (0)203 282 1633
AUS Office & Support: +61 (0) 2 8417 2339

543 NW York Drive, Suite 100, Bend, OR 97703

LinkedIn <http://www.linkedin.com/company/manzama> | Twitter
<https://twitter.com/ManzamaInc> | Facebook
<http://www.facebook.com/manzamainc> | YouTube
<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw>


On Wed, Sep 4, 2019 at 1:43 AM Toke Eskildsen <toes@kb.dk> wrote:

> On Tue, 2019-09-03 at 12:35 -0700, Russell Bahr wrote:
> > Also, if it helps, the content on each server is between around 6.2Gb
> > and 7.8Gb.
>
> We're still missing something here. The trivial query
>
>
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true
> on such a modest index size & document count should not take seconds to
> complete. So we must widen our questioning:
>
> * What is your Xmx for the Solrs?
> (If you use most of the physical memory for Java, there might be too
> little left for OS-caching)
>
> * Do you have a large number of stored or docValued fields?
> (Solr 8 returns docValued fields per default while Solr 4 does not)
>
> * How large is a response? One simple way to check is
> curl '
>
> http://solr.obscured.com:8990/solr/content/select?q=*%3A*&wt=json&indent=true'
>
> | wc -c
> (Solr 4 and 8 compress stored content differently and a very large
> result set requires more CPU power to uncompress in Solr 8 (but less
> IO))
>
> * Do you have any response related defaults in your solrconfig.xml,
> such as faceting or grouping?
> (You might be doing heavy aggregation even if you don't explicitly ask
> for it)
>
>
> - Toke Eskildsen, Royal Danish Library
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message