lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Modassar Ather <modather1...@gmail.com>
Subject Re: Query time out. Solr node goes down.
Date Tue, 18 Aug 2015 09:06:03 GMT
So Toke/Daniel is the node showing *gone* on Solr cloud dashboard is
because of GC pause and it is actually not gone but the ZK is not able to
get the correct state?
The issue is caused by a huge query with many wildcards and phrases in it.
If you see I have mentioned about (*The request took too long to iterate
over terms.). *So does it mean that the terms which are getting expanded
has taken the amount of memory? Just trying to understand what consumes so
much of memory.
I am trying to reproduce the OOM by executing multiple queries in parallel
but not able to whereas I am seeing the memory usage going up by more than
90+% for Solr JVM. So what happens to the query which is executed in
parallel. Do they wait for such query to timeout/complete which is taking
lot of time and resources?
We also have migration to java 8 on our things to do list and will try with
different GC settings.



On Tue, Aug 18, 2015 at 2:08 PM, Daniel Collins <danwcollins@gmail.com>
wrote:

> Ah ok, its ZK timeout then
> (org.apache.zookeeper.KeeperException$SessionExpiredException)
> which is because of your GC pause.
>
> The page Shawn mentioned earlier has several links on how to investigate GC
> issues and some common GC settings, sounds like you need to tweak those.
>
> Generally speaking, I believe Java 8 is considered better for GC
> performance than 7, so you probably want to investigate that.  GC tuning is
> very dependent on the load on your system. You may be running close yo the
> limit under normal load, and that 1 big query is enough to tip it over the
> edge.  We have seen similar issues from time to time. We are still running
> an older Java 7 build with G1GC which we found worked well for us (though
> CMS seems to be the general consensus on the list here), migrating to Java
> 8 is on our "list of things to do", so our settings are probably not that
> relevant.
>
>
> On 18 August 2015 at 09:04, Toke Eskildsen <te@statsbiblioteket.dk> wrote:
>
> > On Tue, 2015-08-18 at 10:38 +0530, Modassar Ather wrote:
> > > Kindly help me understand, even if there is a a GC pause why the solr
> > node
> > > will go down.
> >
> > If a stop-the-world GC is in progress, it is not possible for an
> > external service to know if this is because a GC is in progress or the
> > node is dead. If the GC takes longer than the relevant timeouts, the
> > external conclusion is that it is dead.
> >
> > In you next post you state that there is very heavy GC going on, so it
> > would seem that your main problem is that your heap is too small for
> > your setup.
> >
> > Getting OOM for a 200GB index with 24GB heap is not at all impossible,
> > but it is a bit of a red flag. If you have very high values for your
> > caches or perform faceting on a lot of different fields, that might be
> > the cause. If you describe your setup in more detail, we might be able
> > to help find the cause for your relatively high heap requirement.
> >
> > - Toke Eskildsen, State and University Library, Denmark
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message