lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Timeout occured while waiting response from server at: http://***/solr/commodityReview
Date Wed, 02 Nov 2016 17:10:15 GMT
You need to move to SolrCloud when it's
time to shard ;).....

More seriously, at some point simply adding more
memory will not be adequate. Either your JVM
heap will to grow to a point where you start encountering
GC pauses or the time to serve requests will
increase unacceptably. "when?" you ask? well
unfortunately there are no guidelines that can be
guaranteed, here's a long blog on the subject:

https://lucidworks.com/blog/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

The short form is you need to stress-test your
index and query patterns.

Now, I've seen 20M docs strain a 32G Java heap. I've
seen 300M docs give very nice response times with
12G of memory. It Depends (tm).

Whether to put Solr on bare metal or not: There's
inevitably some penalty for a VM. That said there are lots
of places that use VMs successfully. Again, stress
testing is the key.

And finally, using docValues for any field that sorts,
facets or groups will reduce the JVM requirements
significantly, albeit by using OS memory space, see
Uwe's excellent blog:

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Best,
Erick

On Tue, Nov 1, 2016 at 10:23 PM, Kent Mu <solr.study@gmail.com> wrote:
> Thanks, I got it, Erick!
>
> the size of our index data is more than 30GB every year now, and it is
> still growing up, and actually our solr now is running on a virtual
> machine. so I wonder if we need to deploy solr in a physical machine, or I
> can just upgrade the physical memory of our Virtual machines?
>
> Best,
> Kent
>
> 2016-11-02 11:33 GMT+08:00 Erick Erickson <erickerickson@gmail.com>:
>
>> Kent: OK, I see now. Then a minor pedantic point...
>>
>> It'll avoid confusion if you use master and slaves
>> rather than master and replicas when talking about
>> non-cloud setups.
>>
>> The equivalent in SolrCloud is leader and replicas.
>>
>> No big deal either way, just FYI.
>>
>> Best,
>> Erick
>>
>> On Tue, Nov 1, 2016 at 8:09 PM, Kent Mu <solr.study@gmail.com> wrote:
>> > Thanks a lot for your reply, Shawn!
>> >
>> > no other applications on the server, I agree with you that we need to
>> > upgrade physical memory, and allocate the reasonable jvm size, so that
>> the
>> > operating system have spare memory available to cache the index.
>> >
>> > actually, we have nearly 100 million of data every year now, and it is
>> > still growing, and actually our solr now is running on a virtual machine.
>> > so I wonder if we need to deploy solr in a physical machine.
>> >
>> > Best Regards!
>> > Kent
>> >
>> > 2016-11-01 21:18 GMT+08:00 Shawn Heisey <apache@elyograg.org>:
>> >
>> >> On 11/1/2016 1:07 AM, Kent Mu wrote:
>> >> > Hi friends! We come across an issue when we use the solrj(4.9.1) to
>> >> > connect to solr server, our deployment is one master with 10 replicas.
>> >> > we index data to the master, and search data from the replicas via
>> >> > load balancing. the error stack is as below: *Timeout occured while
>> >> > waiting response from server at:
>> >> > http://review.solrsearch3.cnsuning.com/solr/commodityReview
>> >> > <http://review.solrsearch3.cnsuning.com/solr/commodityReview>*
>> >> > org.apache.solr.client.solrj.SolrServerException: Timeout occured
>> >> > while waiting response from server at:
>> >>
>> >> This shows that you are connecting to port 80.  It is relatively rare to
>> >> run Solr on port 80, though it is possible.  Do you have an intermediate
>> >> layer, like a proxy or a load balancer?  If so, you'll need to ensure
>> >> that there's not a problem there.  If it works normally when replication
>> >> isn't happening, that's probably not a worry.
>> >>
>> >> > It takes place not often. after analysis, we find that only when the
>> >> > replicas Synchronous Data from master solr server. it seem that when
>> >> > the replicas block search requests when synchronizing data from
>> >> > master, is that true?
>> >>
>> >> Solr should be able to continue serving requests while replication
>> >> happens.  I have never heard of this happening before, and I never ran
>> >> into it when I was using replication a long time ago on version 1.4.x.
>> >> I think it is more likely that you've got a memory issue than a bug.  If
>> >> it IS a bug, it will *not* be fixed in a 4.x version, you would need to
>> >> upgrade to 6.x and see whether it's still a problem.  Version 6.2.1 is
>> >> the latest at the moment, and release plans are underway for 6.3 right
>> now.
>> >>
>> >> > I wonder if it is because that our solr server hardware configuration
>> >> > is too low? the physical memory is 8G with 4 cores. and the JVM we
set
>> >> > is Xms512m, Xmx7168m.
>> >>
>> >> The following assumes that there is no other software on the server,
>> >> like a database, or an application server, web server, etc.  If there
>> >> is, any issues are likely to be a result of extreme memory starvation,
>> >> and possibly swapping.  Additional physical memory is definitely needed
>> >> if there is other software on the server beyond basic OS tools.
>> >>
>> >> If the total index data that is on your server is larger than about 1.5
>> >> to 2GB, chances are excellent that you do not have enough free memory to
>> >> cache that data effectively, which can lead to major performance
>> >> issues.  You've only left about 1GB of memory in the system for that
>> >> purpose, and that memory must also run the entire operating system,
>> >> which can take a significant percentage of 1GB.  With a large index, I
>> >> would strongly recommend adding memory to this server.
>> >>
>> >> https://wiki.apache.org/solr/SolrPerformanceProblems
>> >>
>> >> As mentioned in that wiki page, for good performance Solr absolutely
>> >> requires that the operating system have spare memory available to cache
>> >> the index.  In general, allocating almost all your memory to the Java
>> >> heap is a bad idea with Solr.
>> >>
>> >> If your index *is* smaller than 1.5 to 2GB, allocating a 7GB heap is
>> >> probably not necessary, unless you are doing *incredibly* memory-hungry
>> >> queries, such as grouping, faceting, or sorting on many fields.  If you
>> >> can reduce the heap size, there would be more memory available for
>> caching.
>> >>
>> >> Indexing can sometimes cause very large merges to happen, and a full
>> >> index optimize would rewrite the entire index.  Replication copies the
>> >> changed index files, and if the size of the changes is significant,
>> >> additional memory can be required for good performance.  See the special
>> >> note on the wiki page above about optimizes.
>> >>
>> >> Thanks,
>> >> Shawn
>> >>
>> >>
>>

Mime
View raw message