lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Solr Trunk Heap Space Issues
Date Thu, 01 Oct 2009 19:37:56 GMT
Whoops. There is my lazy brain for you - march, may, august - all the
same ;)

Okay - forgot Solr went straight down and used FieldSortedHitQueue.

So it all still makes sense ;)

Still interested in seeing his field sanity output to see whats possibly
being doubled.

Yonik Seeley wrote:
> On Thu, Oct 1, 2009 at 3:14 PM, Mark Miller <markrmiller@gmail.com> wrote:
>   
>> bq. Tons of changes since... including the per-segment
>> searching/sorting/function queries (I think).
>>
>> Yup. I actually didn't think so, because that was committed to Lucene in
>> Feburary - but it didn't come into Solr till March 10th. March 5th just
>> ducked it.
>>     
>
> Jeff said May 5th
>
> But it wasn't until the end of May that Solr started using Lucene's
> new sorting facilities that worked per-segment.
>
> -Yonik
> http://www.lucidimagination.com
>
>
>   
>> Yonik Seeley wrote:
>>     
>>> On Thu, Oct 1, 2009 at 11:41 AM, Jeff Newburn <jnewburn@zappos.com> wrote:
>>>
>>>       
>>>> I am trying to update to the newest version of solr from trunk as of May
>>>> 5th.
>>>>
>>>>         
>>> Tons of changes since... including the per-segment
>>> searching/sorting/function queries (I think).
>>>
>>> Do you sort on any single valued fields that you also facet on?
>>> Do you use ord() or rord() in any function queries?
>>>
>>> Unfortunately, some of these things will take up more memory because
>>> some things still cache FieldCache elements with the top-level reader,
>>> while some use segment readers.  The direction is going toward all
>>> segment readers, but we're not there yet (and won't be for 1.4).
>>> ord() rord() will never be fixed... people need to migrate to
>>> something else.
>>>
>>> http://issues.apache.org/jira/browse/SOLR-1111 is the main issue for this.
>>>
>>> If course, I've really only been talking about search related changes.
>>>  Nothing on the indexing side should cause greater memory usage....
>>> but perhaps the indexing side could run out of memory due to the
>>> search side taking up more.
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>>       
>>>>  I updated and compiled from trunk as of yesterday (09/30/2009).  When
>>>> I try to do a full import I am receiving a GC heap error after changing
>>>> nothing in the configuration files.  Why would this happen in the most
>>>> recent versions but not in the version from a few months ago.  The stack
>>>> trace is below.
>>>>
>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.update.processor.LogUpdateProcessor
>>>> finish
>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, 167353,
>>>> ...(83 more)]} 0 35991
>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log
>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>    at java.util.Arrays.copyOfRange(Arrays.java:3209)
>>>>    at java.lang.String.<init>(String.java:215)
>>>>    at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384)
>>>>    at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
>>>>    at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280)
>>>>    at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139)
>>>>    at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>>>>    at
>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt
>>>> reamHandlerBase.java:54)
>>>>    at
>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
>>>> java:131)
>>>>    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
>>>>    at
>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
>>>> 38)
>>>>    at
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
>>>> 241)
>>>>    at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
>>>> FilterChain.java:235)
>>>>    at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
>>>> ain.java:206)
>>>>    at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
>>>> va:233)
>>>>    at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
>>>> va:175)
>>>>    at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
>>>> )
>>>>    at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
>>>> )
>>>>    at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
>>>> :109)
>>>>    at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
>>>>    at
>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
>>>> 879)
>>>>    at
>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H
>>>> ttp11NioProtocol.java:719)
>>>>    at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:
>>>> 2080)
>>>>    at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
>>>> va:886)
>>>>    at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>>>> 08)
>>>>    at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute
>>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 QTime=5265
>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log
>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>>
>>>> --
>>>> Jeff Newburn
>>>> Software Engineer, Zappos.com
>>>> jnewburn@zappos.com - 702-943-7562
>>>>
>>>>
>>>>
>>>>         
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>>     


-- 
- Mark

http://www.lucidimagination.com




Mime
View raw message