lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Java heap space exception in 4.2.1
Date Mon, 27 May 2013 11:55:58 GMT
400M docs is quite a large number of documents for a single piece of
hardware, and
if you're faceting over a large number of unique values, this will
chew up memory.

So it's not surprising that you're seeing OOMs, I suspect you just have too many
documents on a single machine..

Best
Erick


On Mon, May 27, 2013 at 3:11 AM, Jam Luo <cooljam2008@gmail.com> wrote:
> I am sorry about a type mistake  8,000,000,000 -> 800,000,000
>
>
> 2013/5/27 Jam Luo <cooljam2008@gmail.com>
>
>> I have the same problem. at 4.1  ,a solr instance could take 8,000,000,000
>> doc. but at 4.2.1, a instance only take 400,000,000 doc, it will oom at
>> facet query.  the facet field was token by space.
>>
>> May 27, 2013 11:12:55 AM org.apache.solr.common.SolrException log
>> SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java
>> heap space
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1338)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
>>         at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
>>         at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
>>         at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
>>         at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
>>         at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
>>         at org.eclipse.jetty.server.Server.handle(Server.java:350)
>>         at
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
>>         at
>> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
>>         at
>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
>>         at
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
>>         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851)
>>         at
>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>         at
>> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
>>         at
>> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
>>         at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
>>         at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
>>         at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>         at
>> org.apache.lucene.index.DocTermOrds.uninvert(DocTermOrds.java:448)
>>         at
>> org.apache.solr.request.UnInvertedField.<init>(UnInvertedField.java:179)
>>         at
>> org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:664)
>>         at
>> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:426)
>>         at
>> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:517)
>>         at
>> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:252)
>>         at
>> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:78)
>>         at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
>>         at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1825)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
>>         at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1338)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
>>         at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
>>         at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
>>         at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
>>         at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
>>         at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>>         at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
>>         at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
>>         at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
>>         at org.eclipse.jetty.server.Server.handle(Server.java:350)
>>         at
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
>>         at
>> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
>>         at
>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
>>         at
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
>>         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851)
>>
>>
>>
>> 2013/5/19 J Mohamed Zahoor <zahoor@indix.com>
>>
>>>
>>> aah… was doing a facet on a double field which was having 6 decimal
>>> places…
>>> No surprise that the lucene cache got full…
>>>
>>> .z/ahoor
>>>
>>> On 17-May-2013, at 11:56 PM, J Mohamed Zahoor <zahoor@indix.com> wrote:
>>>
>>> > Memory increase a lot with queries which have facets…
>>> >
>>> >
>>> > ./Zahoor
>>> >
>>> >
>>> > On 17-May-2013, at 10:00 PM, Shawn Heisey <solr@elyograg.org> wrote:
>>> >
>>> >> On 5/17/2013 1:17 AM, J Mohamed Zahoor wrote:
>>> >>> I moved to 4.2.1 from 4.1 recently.. everything was working fine
>>> until i added few more stats query..
>>> >>> Now i am getting this error frequently that solr does not run even
>>> for 2 minutes continuously.
>>> >>> All 5GB is getting used instantaneously in few queries...
>>> >>
>>> >> Someone on IRC ran into memory problems upgrading from 4.0 to 4.2. 
It
>>> >> wasn't OOM errors, they were just using a lot more heap than they were
>>> >> before and running into constant full garbage collections.
>>> >>
>>> >> There is another message on this list about someone who upgraded from
>>> >> 3.5 to 4.2 and is having memory troubles.
>>> >>
>>> >> The person on IRC made most of their fields unstored and reindexed,
>>> >> which fixed the problem for them.  They only needed a few fields
>>> stored.
>>> >>
>>> >> Because the IRC user was on 4.0, I originally thought it had something
>>> >> to do with compressed stored fields, but on this thread, they started
>>> >> with 4.1.  If that was the released 4.1.0 and not a SNAPSHOT version,
>>> >> then they had compressed stored fields before the upgrade.
>>> >>
>>> >> The user on IRC was not using termvectors or docvalues, which would
be
>>> >> potential pain points unique to 4.2.
>>> >>
>>> >> I'm using 4.2.1 with no trouble in my setup, but I do have a heap
>>> that's
>>> >> considerably larger than I need.  There are no apparent memory leaks
-
>>> >> it's been running for over a month with updates once a minute.  I've
>>> >> finally switched over from the 3.5.0 index to the new one, so for the
>>> >> last few days, it has been also taking our full query load.
>>> >>
>>> >> What could have changed between 4.1 and 4.2 to cause dramatically
>>> >> increased memory usage?
>>> >>
>>> >> From my /admin/system:
>>> >>
>>> >> <date name="startTime">2013-04-05T15:52:55.751Z</date>
>>> >>
>>> >> Thanks,
>>> >> Shawn
>>> >>
>>> >
>>>
>>>
>>

Mime
View raw message