lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ariel <ionat...@gmail.com>
Subject Re: Solr and Garbage Collection
Date Fri, 25 Sep 2009 16:07:16 GMT
You are saying that I should give more memory than 12GB?
When I was with 10GB I had the exceptions that I sent. Switching to 12GB
made them disappear.
So I think I don't have problems with FieldCache any more. What it seems
like a problem is 11% on the application time dedicated to GC. Specially
when those servers are under really heavy load.
I think that's why I sometimes get queries that in one moment are being
executed in a few ms and a moment after 20 seconds!

It seems like I should tune my jvm, don't you think so?

On Fri, Sep 25, 2009 at 1:01 PM, Fuad Efendi <fuad@efendi.ca> wrote:

> Give it even more memory.
>
> Lucene FieldCache is used to store non-tokenized single-value non-boolean
> (DocumentId -> FieldValue) pairs, and it is used (in-full!) for instance
> for
> sorting query results.
>
> So that if you have 100,000,000 documents with specific heavily distributed
> field values (cardinality is high! Size is 100bytes!) you need
> 10,000,000,000 bytes for just this instance of FieldCache.
>
> GC does not play any role. FieldCache won't be GC-collected.
>
>
> -Fuad
> http://www.linkedin.com/in/liferay
>
>
>
> > -----Original Message-----
> > From: Jonathan Ariel [mailto:ionathan@gmail.com]
> > Sent: September-25-09 11:37 AM
> > To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> > Subject: Re: Solr and Garbage Collection
> >
> > Right, now I'm giving it 12GB of heap memory.
> > If I give it less (10GB) it throws the following exception:
> >
> > Sep 5, 2009 7:18:32 PM org.apache.solr.common.SolrException log
> > SEVERE: java.lang.OutOfMemoryError: Java heap space
> >         at
> >
>
> org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:3
> 61
> > )
> >         at
> > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
> >         at
> >
>
> org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:3
> 52
> > )
> >         at
> >
>
> org.apache.solr.request.SimpleFacets.getFieldCacheCounts(SimpleFacets.java:2
> 67
> > )
> >         at
> > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:185)
> >         at
> >
>
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:2
> 07
> > )
> >         at
> >
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:104)
> >         at
> >
>
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java
> :7
> > 0)
> >         at
> >
>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHand
> le
> > r.java:169)
> >         at
> >
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
> ja
> > va:131)
> >         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
> >         at
> >
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
> 03
> > )
> >         at
> >
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 23
> > 2)
> >         at
> >
>
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler
> .j
> > ava:1089)
> >         at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> >         at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> >         at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> >         at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> >         at
> > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
> >         at
> >
>
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerColl
> ec
> > tion.java:211)
> >         at
> >
>
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:11
> 4)
> >         at
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
> >         at org.mortbay.jetty.Server.handle(Server.java:285)
> >         at
> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
> >         at
> >
>
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:
> 83
> > 5)
> >         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:641)
> >         at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
> >         at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> >         at
> >
>
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:22
> 6)
> >         at
> >
>
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:4
> 42
> > )
> >
> > On Fri, Sep 25, 2009 at 10:55 AM, Yonik Seeley
> > <yonik@lucidimagination.com>wrote:
> >
> > > On Fri, Sep 25, 2009 at 9:30 AM, Jonathan Ariel <ionathan@gmail.com>
> > > wrote:
> > > > Hi to all!
> > > > Lately my solr servers seem to stop responding once in a while. I'm
> using
> > > > solr 1.3.
> > > > Of course I'm having more traffic on the servers.
> > > > So I logged the Garbage Collection activity to check if it's because
> of
> > > > that. It seems like 11% of the time the application runs, it is
> stopped
> > > > because of GC. And some times the GC takes up to 10 seconds!
> > > > Is is normal? My instances run on a 16GB RAM, Dual Quad Core Intel
> Xeon
> > > > servers. My index is around 10GB and I'm giving to the instances 10GB
> of
> > > > RAM.
> > >
> > > Bigger heaps lead to bigger GC pauses in general.
> > > Do you mean that you are giving the JVM a 10GB heap?  Were you getting
> > > OOM exceptions with a smaller heap?
> > >
> > > -Yonik
> > > http://www.lucidimagination.com
> > >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message