Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4E47E359.4010803@transaxtions.com>
Date: Sun, 14 Aug 2011 08:01:45 -0700
From: Nagendra Nagarajayya <nnagarajayya@transaxtions.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US;
 rv:1.9.2.18) Gecko/20110616 Lightning/1.0b2 Thunderbird/3.1.11
MIME-Version: 1.0
To: solr-user@lucene.apache.org
Subject: Re: exceeded limit of maxWarmingSearchers ERROR
References: 
 <CAL8Qe1MQp-yxktpSbmv9moLMZn1FEd13bPx_accY9Ve9SotRoA@mail.gmail.com>
In-Reply-To: 
 <CAL8Qe1MQp-yxktpSbmv9moLMZn1FEd13bPx_accY9Ve9SotRoA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Naveen:

You should try NRT with Apache Solr 3.3 and RankingAlgorithm. You can 
update 10,000 documents / sec while also concurrently searching. You can 
set commit  freq to about 15 mins or as desired. The 10,000 document 
update performance is with the MBArtists index on a dual core Linux 
system. So you may be able to see similar performance on your system. 
You can get more details of the NRT implementation from here:

http://solr-ra.tgels.org/wiki/en/Near_Real_Time_Search_ver_3.x

You can download Apache Solr 3.3 with RankingAlgorithm from here:

http://solr-ra.tgels.org/

(There are no changes to your existing setup, everything should work as 
earlier except for adding the <realtime> tag to your solrconfig.xml)

Regards

- Nagendra Nagarajayya
http://solr-ra.tgels.org
http://rankingalgorithm.tgels.org


On 8/13/2011 6:47 PM, Naveen Gupta wrote:
> Hi,
>
> Most of the settings are default.
>
> We have single node (Memory 1 GB, Index Size 4GB)
>
> We have a requirement where we are doing very fast commit. This is kind of
> real time requirement where we are polling many threads from third party and
> indexes into our system.
>
> We want these results to be available soon.
>
> We are committing for each user (may have 10k threads and inside that 1
> thread may have 10 messages). So overall documents per user will be having
> around .1 million (100000)
>
> Earlier we were using commit Within  as 10 milliseconds inside the document,
> but that was slowing the indexing and we were not getting any error.
>
> As we removed the commit Within, indexing became very fast. But after that
> we started experiencing in the system
>
> As i read many forums, everybody told that this is happening because of very
> fast commit rate, but what is the solution for our problem?
>
> We are using CURL to post the data and commit
>
> Also till now we are using default solrconfig.
>
> Aug 14, 2011 12:12:04 AM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher.
> exceeded limit of maxWarmingSearchers=2, try again later.
>          at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1052)
>          at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:424)
>          at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>          at
> org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
>          at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
>          at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:55)
>          at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>          at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
>          at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>          at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
>          at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>          at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>          at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>          at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>          at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>          at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>          at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>          at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>          at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
>          at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
>          at
> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>          at java.lang.Thread.run(Thread.java:662)
>