lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Yingcai Sun <Y...@vizrt.com>
Subject Re: Solr index slow response
Date Mon, 18 Mar 2019 13:36:07 GMT
Hi, Emir,

My system used to run with max 32GB, the response time is bad as well.  swap is set to 4GB,
there 3.2 free, I doubt swap would affect it since there is such huge free memory.

I could try to with set Xms and Xmx to the same value, but I doubt how much would that change
the response time.


BRs

//Aaron

________________________________
From: Emir Arnautović <emir.arnautovic@sematext.com>
Sent: Monday, March 18, 2019 2:19:19 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr index slow response

Hi Aaron,
Without looking too much into numbers, my bet would be that it is large heap that is causing
issues. I would decrease is significantly (<30GB) and see if it is enough for your max
load. Also, disable swap or reduce swappiness to min.

In any case, you should install some monitoring tool that would help you do better analysis
when you run into problems. One such tool is our monitoring solution: https://sematext.com/spm

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 18 Mar 2019, at 13:14, Aaron Yingcai Sun <YSun@vizrt.com> wrote:
>
> Hello, Emir,
>
> Thanks for the reply, this is the solr version and heap info, standalone single solr
server. I don't have monitor tool connected. only look at 'top', has not seen cpu spike so
far, when the slow response happens, cpu usage is not high at all, around 30%.
>
>
> # curl 'http://.../solr/admin/info/system?wt=json&indent=true'
> {
>  "responseHeader":{
>    "status":0,
>    "QTime":27},
>  "mode":"std",
>  "solr_home":"/ardome/solr",
>  "lucene":{
>    "solr-spec-version":"6.5.1",
>    "solr-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - jimczi - 2017-04-21
12:23:42",
>    "lucene-spec-version":"6.5.1",
>    "lucene-impl-version":"6.5.1 cd1f23c63abe03ae650c75ec8ccb37762806cc75 - jimczi - 2017-04-21
12:17:15"},
>  "jvm":{
>    "version":"1.8.0_144 25.144-b01",
>    "name":"Oracle Corporation Java HotSpot(TM) 64-Bit Server VM",
>    "spec":{
>      "vendor":"Oracle Corporation",
>      "name":"Java Platform API Specification",
>      "version":"1.8"},
>    "jre":{
>      "vendor":"Oracle Corporation",
>      "version":"1.8.0_144"},
>    "vm":{
>      "vendor":"Oracle Corporation",
>      "name":"Java HotSpot(TM) 64-Bit Server VM",
>      "version":"25.144-b01"},
>    "processors":32,
>    "memory":{
>      "free":"69.1 GB",
>      "total":"180.2 GB",
>      "max":"266.7 GB",
>      "used":"111 GB (%41.6)",
>      "raw":{
>        "free":74238728336,
>        "total":193470136320,
>        "max":286331502592,
>        "used":119231407984,
>        "used%":41.64103736566334}},
>    "jmx":{
>      "bootclasspath":"/usr/java/jdk1.8.0_144/jre/lib/resources.jar:/usr/java/jdk1.8.0_144/jre/lib/rt.jar:/usr/java/jdk1.8.0_144/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_144/jre/lib/jsse.jar:/usr/java/jdk1.8.0_144/jre/lib/jce.jar:/usr/java/jdk1.8.0_144/jre/lib/charsets.jar:/usr/java/jdk1.8.0_144/jre/lib/jfr.jar:/usr/java/jdk1.8.0_144/jre/classes",
>      "classpath":"...",
>      "commandLineArgs":["-Xms100G",
>        "-Xmx300G",
>        "-DSTOP.PORT=8079",
>        "-DSTOP.KEY=..",
>        "-Dsolr.solr.home=..",
>        "-Djetty.port=8983"],
>      "startTime":"2019-03-18T09:35:27.892Z",
>      "upTimeMS":9258422}},
>  "system":{
>    "name":"Linux",
>    "arch":"amd64",
>    "availableProcessors":32,
>    "systemLoadAverage":14.72,
>    "version":"3.0.101-311.g08a8a9d-default",
>    "committedVirtualMemorySize":2547960700928,
>    "freePhysicalMemorySize":4530696192,
>    "freeSwapSpaceSize":3486846976,
>    "processCpuLoad":0.3257436126790475,
>    "processCpuTime":93869450000000,
>    "systemCpuLoad":0.3279781055816521,
>    "totalPhysicalMemorySize":406480175104,
>    "totalSwapSpaceSize":4302303232,
>    "maxFileDescriptorCount":32768,
>    "openFileDescriptorCount":385,
>    "uname":"Linux ... 3.0.101-311.g08a8a9d-default #1 SMP Wed Dec 14 10:15:37 UTC 2016
(08a8a9d) x86_64 x86_64 x86_64 GNU/Linux\n",
>    "uptime":" 13:09pm  up 5 days 21:23,  7 users,  load average: 14.72, 12.28, 11.48\n"}}
>
>
>
>
> ________________________________
> From: Emir Arnautović <emir.arnautovic@sematext.com>
> Sent: Monday, March 18, 2019 12:10:30 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr index slow response
>
> Hi Aaron,
> Which version of Solr? How did you configure your heap? Is it standalone Solr or SolrCloud?
A single server? Do you use some monitoring tool? Do you see some spikes, pauses or CPU usage
is constant?
>
> Thanks,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 18 Mar 2019, at 11:47, Aaron Yingcai Sun <YSun@vizrt.com> wrote:
>>
>> Hello, Solr!
>>
>>
>> We are having some performance issue when try to send documents for solr to index.
The repose time is very slow and unpredictable some time.
>>
>>
>> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, while 300 GB
is reserved for solr, while this happening, cpu usage is around 30%, mem usage is 34%.  io
also look ok according to iotop. SSD disk.
>>
>>
>> Our application send 100 documents to solr per request, json encoded. the size is
around 5M each time. some times the response time is under 1 seconds, some times could be
300 seconds, the slow response happens very often.
>>
>>
>> "Soft AutoCommit: disabled", "Hard AutoCommit: if uncommited for 3600000ms; if 1000000
uncommited docs"
>>
>>
>> There are around 100 clients sending those documents at the same time, but each for
the client is blocking call which wait the http response then send the next one.
>>
>>
>> I tried to make the number of documents smaller in one request, such as 20, but 
still I see slow response time to time, like 80 seconds.
>>
>>
>> Would you help to give some hint how improve the response time?  solr does not seems
very loaded, there must be a way to make the response faster.
>>
>>
>> BRs
>>
>> //Aaron
>>
>>
>>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message