lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Yingcai Sun <Y...@vizrt.com>
Subject Re: Solr index slow response
Date Mon, 18 Mar 2019 13:30:24 GMT
I'm a bit confused, why large heap size would make it slower?  Isn't that give it enough room
to make it not busy doing GC all the time?

My http/json request contains 100 documents, the total size of the 100 documents is around
5M, there are ~100 client sending those requests continuously.

Previously the JVM is set to max 32 GB ,  the speed was even worse,  now it's running with
min 100GB, max 300GB, it use around 100GB.


this page suggest to use smaller number of documents per request,   https://wiki.apache.org/solr/SolrPerformanceProblems

SolrPerformanceProblems - Solr Wiki<https://wiki.apache.org/solr/SolrPerformanceProblems>
wiki.apache.org
General information. There is a performance bug that makes *everything* slow in versions 6.4.0
and 6.4.1. The problem is fixed in 6.4.2. It is described by SOLR-10130.This is highly version
specific, so if you are not running one of the affected versions, don't worry about it.

So I try to reduce the number, still I could get lots of large response QTime:

190318-142652.695-160214 DBG1:doc_count: 10 , doc_size: 609  KB, Res code: 200, QTime: 47918
ms, Request time: 47921 ms.
190318-142652.704-160179 DBG1:doc_count: 10 , doc_size: 568  KB, Res code: 200, QTime: 36919
ms, Request time: 36922 ms.
190318-142652.780-160197 DBG1:doc_count: 10 , doc_size: 609  KB, Res code: 200, QTime: 36082
ms, Request time: 36084 ms.
190318-142652.859-160200 DBG1:doc_count: 10 , doc_size: 569  KB, Res code: 200, QTime: 36880
ms, Request time: 36882 ms.
190318-142653.131-160148 DBG1:doc_count: 10 , doc_size: 608  KB, Res code: 200, QTime: 37222
ms, Request time: 37224 ms.
190318-142653.154-160211 DBG1:doc_count: 10 , doc_size: 541  KB, Res code: 200, QTime: 37241
ms, Request time: 37243 ms.
190318-142653.223-163490 DBG1:doc_count: 10 , doc_size: 589  KB, Res code: 200, QTime: 37174
ms, Request time: 37176 ms.
190318-142653.359-160154 DBG1:doc_count: 10 , doc_size: 592  KB, Res code: 200, QTime: 37008
ms, Request time: 37011 ms.
190318-142653.497-163491 DBG1:doc_count: 10 , doc_size: 583  KB, Res code: 200, QTime: 24828
ms, Request time: 24830 ms.
190318-142653.987-160208 DBG1:doc_count: 10 , doc_size: 669  KB, Res code: 200, QTime: 23900
ms, Request time: 23902 ms.
190318-142654.114-160208 DBG1:doc_count: 10 , doc_size: 544  KB, Res code: 200, QTime: 121
ms, Request time: 122 ms.
190318-142654.233-160208 DBG1:doc_count: 10 , doc_size: 536  KB, Res code: 200, QTime: 113
ms, Request time: 115 ms.
190318-142654.354-160208 DBG1:doc_count: 10 , doc_size: 598  KB, Res code: 200, QTime: 116
ms, Request time: 117 ms.
190318-142654.466-160208 DBG1:doc_count: 10 , doc_size: 546  KB, Res code: 200, QTime: 107
ms, Request time: 108 ms.
190318-142654.586-160208 DBG1:doc_count: 10 , doc_size: 566  KB, Res code: 200, QTime: 114
ms, Request time: 115 ms.
190318-142654.687-160208 DBG1:doc_count: 10 , doc_size: 541  KB, Res code: 200, QTime: 96
ms, Request time: 98 ms.
190318-142654.768-160208 DBG1:doc_count: 10 , doc_size: 455  KB, Res code: 200, QTime: 75
ms, Request time: 77 ms.
190318-142654.870-160208 DBG1:doc_count: 10 , doc_size: 538  KB, Res code: 200, QTime: 97
ms, Request time: 98 ms.
190318-142654.967-160208 DBG1:doc_count: 10 , doc_size: 539  KB, Res code: 200, QTime: 92
ms, Request time: 93 ms.
190318-142655.096-160208 DBG1:doc_count: 10 , doc_size: 672  KB, Res code: 200, QTime: 124
ms, Request time: 125 ms.
190318-142655.210-160208 DBG1:doc_count: 10 , doc_size: 605  KB, Res code: 200, QTime: 108
ms, Request time: 110 ms.
190318-142655.304-160208 DBG1:doc_count: 10 , doc_size: 481  KB, Res code: 200, QTime: 89
ms, Request time: 90 ms.
190318-142655.410-160208 DBG1:doc_count: 10 , doc_size: 468  KB, Res code: 200, QTime: 101
ms, Request time: 102 ms.


________________________________
From: Toke Eskildsen <toes@kb.dk>
Sent: Monday, March 18, 2019 2:13:12 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr index slow response

On Mon, 2019-03-18 at 10:47 +0000, Aaron Yingcai Sun wrote:
> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM,
> while 300 GB is reserved for solr, [...]

300GB for Solr sounds excessive.

> Our application send 100 documents to solr per request, json encoded.
> the size is around 5M each time. some times the response time is
> under 1 seconds, some times could be 300 seconds, the slow response
> happens very often.
> ...
> There are around 100 clients sending those documents at the same
> time, but each for the client is blocking call which wait the http
> response then send the next one.

100 clients * 5MB/batch = at most 500MB. Or maybe you meant 100 clients
* 100 documents * 5MB/document = at most 50GB? Either way it is a long
way from 300GB and the stats you provide further down the thread
indicates that you are overprovisioning quite a lot:

"memory":{
      "free":"69.1 GB",
      "total":"180.2 GB",
      "max":"266.7 GB",
      "used":"111 GB (%41.6)",

Intermittent slow response times are a known effect of having large
Java heaps, due to stop-the-world garbage collections.

Try dialing Xmx _way_ down: If your batches are only 5MB each, try
Xmx=20g or less. I know that the stats above says that Solr uses 111GB,
but the JVM has a tendency to expand the heap quite a lot when it is
getting hammered. If you want to check beforehand, you can see how much
memeory is freed from full GCs in the GC-log.

- Toke Eskildsen, Royal Danish Library



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message