lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From adityab <aditya_ba...@yahoo.com>
Subject Solr3.5 Vs Solr4.1 - Help please
Date Thu, 28 Feb 2013 02:51:22 GMT
Hi,
A little history before i tell the actual issue. Please bare with me. 
In Dev lab with a single VM (2vCPU, 4gb RAM, 3Gb to JVM, JBoss5.1,
JDK1.6.0.30) we used Solr 4.1 and indexed 250K documents with each document
of avg size 22Kb - Totla index size is 3.6GB. 
Every thing works good. The only query that will be hitting this cluster is
with *q=masterId:XXX* where masterId is our unique key defined in schema.
All our fields are stored

Now for performance testing we requested our perf. lab to upgrade their Solr
3.5 to Solr 4.1. Performance Lab has 1 Master and 2 slaves. After upgrading
To Solr 4.1 the first thing they observed is index size shrieked to 1/2 of
Solr3.5 (This is good) They have the same machine Spec as Dev. 

As soon as we started hitting the Slaves with queries we saw CPU Spike upto
100% withing 2mins at 15QPS and the Application becomes sluggish to
unresponsive. 
1. The same machine had Solr 3.5 earlier word like charm and we took the QPS
upto 150. But with 4.1 we are stuck below 15. 
2. Even after we stop the load runner we still see CPU 100% for long time. 
3. We also performed same query test by reverting 1 slave to Solr3.5 and
second slave to 4.1. 3.5 had no issue and 4.1 was causing CPU 100% (Master
was turned off so no replication) 
4. Took thread dumps as to check whats consuming CPU. There were no dead
locks and all threads are in Block State. of which most are as seen below
this message. 

Burning ourselves for several hours so far we couldn't identify why 3.5
performs better than 4.1. Where as the only difference is solr and the
related change required for 4.1 in schema and config file. Also we use
MMapdirectory for both. 

Any advice or direction would be great. 

thanks


Thread 32403: (state = BLOCKED)
 - java.util.AbstractList$Itr.hasNext() @bci=8, line=339 (Compiled frame;
information may be imprecise)
 - org.apache.lucene.document.Document.getFields(java.lang.String) @bci=19,
line=175 (Compiled frame)
 - org.apache.lucene.document.LazyDocument$LazyField.stringValue() @bci=38,
line=107 (Compiled frame)
 -
org.apache.solr.schema.FieldType.toExternal(org.apache.lucene.index.IndexableField)
@bci=1, line=330 (Compiled frame)
 -
org.apache.solr.schema.FieldType.toObject(org.apache.lucene.index.IndexableField)
@bci=2, line=339 (Compiled frame)
 -
org.apache.solr.response.BinaryResponseWriter$Resolver.getValue(org.apache.solr.schema.SchemaField,
org.apache.lucene.index.IndexableField) @bci=120, line=223 (Compiled frame)
 -
org.apache.solr.response.BinaryResponseWriter$Resolver.getDoc(org.apache.lucene.document.Document)
@bci=76, line=186 (Compiled frame)
 -
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(org.apache.solr.response.ResultContext,
org.apache.solr.common.util.JavaBinCodec) @bci=205, line=147 (Compiled
frame)
 -
org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(org.apache.solr.response.ResultContext,
org.apache.solr.common.util.JavaBinCodec) @bci=126, line=173 (Compiled
frame)
 -
org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(java.lang.Object,
org.apache.solr.common.util.JavaBinCodec) @bci=13, line=86 (Compiled frame)
 - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object)
@bci=24, line=154 (Compiled frame)
 -
org.apache.solr.common.util.JavaBinCodec.writeNamedList(org.apache.solr.common.util.NamedList)
@bci=53, line=144 (Compiled frame)
 - org.apache.solr.common.util.JavaBinCodec.writeKnownType(java.lang.Object)
@bci=22, line=234 (Compiled frame)
 - org.apache.solr.common.util.JavaBinCodec.writeVal(java.lang.Object)
@bci=2, line=149 (Compiled frame)
 - org.apache.solr.common.util.JavaBinCodec.marshal(java.lang.Object,
java.io.OutputStream) @bci=20, line=92 (Compiled frame)
 - org.apache.solr.response.BinaryResponseWriter.write(java.io.OutputStream,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=68, line=50 (Compiled
frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(org.apache.solr.response.SolrQueryResponse,
javax.servlet.ServletResponse, org.apache.solr.response.QueryResponseWriter,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.servlet.cache.Method) @bci=104, line=397 (Interpreted frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=867, line=282
(Interpreted frame)
 -
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame)
 -
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=61, line=96
(Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=119, line=235 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=206 (Compiled frame)
 -
org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=800, line=235 (Interpreted
frame)
 -
org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=363, line=191 (Compiled frame)
 -
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=386, line=190 (Compiled frame)
 -
org.jboss.web.tomcat.security.JaccContextValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=117, line=92 (Interpreted
frame)
 -
org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response, org.jboss.servlet.http.HttpEvent)
@bci=80, line=126 (Interpreted frame)
 -
org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=4, line=70 (Interpreted frame)
 -
org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=64, line=127 (Interpreted
frame)
 -
org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=6, line=102 (Compiled frame)
 -
org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=35, line=158 (Compiled frame)
 -
org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=42, line=109 (Compiled frame)
 -
org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request,
org.apache.coyote.Response) @bci=157, line=330 (Interpreted frame)
 - org.apache.coyote.ajp.AjpProcessor.process(java.net.Socket) @bci=322,
line=436 (Compiled frame)
 -
org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(java.net.Socket)
@bci=39, line=384 (Interpreted frame)
 - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=447
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)




--
View this message in context: http://lucene.472066.n3.nabble.com/Solr3-5-Vs-Solr4-1-Help-please-tp4043543.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message