lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Murphy <jim.mur...@pobox.com>
Subject Re: Tomcat 6 HTTP Connector Threads all blocked
Date Sun, 01 Mar 2009 21:12:00 GMT

Thanks Yonik,

1. We're using rsync, snappuller/snapinstaller scripts for syncing masters
and slaves.  

2.  Ran jstack and found 2 kinds of stacks blocke on Socket.write0.  The
first is the one I sent previously - and seems difficult to figure the
request type. This one is serializing a SolrDocumentList as JSON which makes
me believe its a query result on the slave side of things.  Our inserts use
XML but our queries use JSON.

I suppose this question might be better posed on the Tomcat list, but I'm
surprised that the server sockets just block if its misbehaving clients.  We
have ruby processes that are the only clients to Solr, so its possible that
there's bugs in ruby's Net:Http.  But is there something I can enable
server-side to make the server more aggresive on closing connections?  

 
The "other" stack trace:

 - java.net.SocketOutputStream.socketWrite0(java.io.FileDescriptor, byte[],
int, int) @bci=0 (Interpreted frame)
 - java.net.SocketOutputStream.socketWrite(byte[], int, int) @bci=44,
line=92 (Interpreted frame)
 - java.net.SocketOutputStream.write(byte[], int, int) @bci=4, line=136
(Interpreted frame)
 - org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(byte[], int,
int) @bci=11, line=737 (Interpreted frame)
 - org.apache.tomcat.util.buf.ByteChunk.flushBuffer() @bci=71, line=434
(Compiled frame)
 - org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int) @bci=159,
line=349 (Compiled frame)
 -
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk,
org.apache.coyote.Response) @bci=31, line=761 (Interpreted frame)
 -
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(org.apache.tomcat.util.buf.ByteChunk,
org.apache.coyote.Response) @bci=107, line=127 (Interpreted frame)
 -
org.apache.coyote.http11.InternalOutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk,
org.apache.coyote.Response) @bci=49, line=570 (Interpreted frame)
 - org.apache.coyote.Response.doWrite(org.apache.tomcat.util.buf.ByteChunk)
@bci=6, line=560 (Compiled frame)
 - org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int,
int) @bci=38, line=353 (Compiled frame)
 - org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int) @bci=77,
line=325 (Compiled frame)
 - org.apache.tomcat.util.buf.IntermediateOutputStream.write(byte[], int,
int) @bci=14, line=242 (Compiled frame)
 - sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Compiled frame)
 - sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Compiled
frame)
 - sun.nio.cs.StreamEncoder.implFlush() @bci=1, line=276 (Compiled frame)
 - sun.nio.cs.StreamEncoder.flush() @bci=12, line=122 (Compiled frame)
 - java.io.OutputStreamWriter.flush() @bci=4, line=212 (Compiled frame)
 - org.apache.tomcat.util.buf.WriteConvertor.flush() @bci=1, line=191
(Compiled frame)
 - org.apache.tomcat.util.buf.C2BConverter.flushBuffer() @bci=4, line=134
(Compiled frame)
 - org.apache.catalina.connector.OutputBuffer.write(char[], int, int)
@bci=22, line=439 (Interpreted frame)
 - org.apache.catalina.connector.CoyoteWriter.write(char[], int, int)
@bci=15, line=143 (Interpreted frame)
 - org.apache.solr.common.util.FastWriter.write(char) @bci=25, line=55
(Compiled frame)
 - org.apache.solr.request.JSONWriter.writeStr(java.lang.String,
java.lang.String, boolean) @bci=60, line=610 (Compiled frame)
 - org.apache.solr.request.TextResponseWriter.writeVal(java.lang.String,
java.lang.Object) @bci=26, line=118 (Compiled frame)
 - org.apache.solr.request.JSONWriter.writeSolrDocument(java.lang.String,
org.apache.solr.common.SolrDocument, java.util.Set, java.util.Map) @bci=148,
line=419 (Compiled frame)
 -
org.apache.solr.request.JSONWriter.writeSolrDocumentList(java.lang.String,
org.apache.solr.common.SolrDocumentList, java.util.Set, java.util.Map)
@bci=240, line=547 (Interpreted frame)
 - org.apache.solr.request.TextResponseWriter.writeVal(java.lang.String,
java.lang.Object) @bci=243, line=147 (Compiled frame)
 -
org.apache.solr.request.JSONWriter.writeNamedListAsMapWithDups(java.lang.String,
org.apache.solr.common.util.NamedList) @bci=70, line=175 (Compiled frame)
 - org.apache.solr.request.JSONWriter.writeNamedList(java.lang.String,
org.apache.solr.common.util.NamedList) @bci=10, line=288 (Interpreted frame)
 - org.apache.solr.request.JSONWriter.writeResponse() @bci=45, line=88
(Interpreted frame)
 - org.apache.solr.request.JSONResponseWriter.write(java.io.Writer,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.request.SolrQueryResponse) @bci=14, line=49 (Interpreted
frame)
 -
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=767, line=257
(Interpreted frame)
 -
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=117, line=235 (Interpreted frame)
 -
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=206 (Interpreted frame)
 -
org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=804, line=233 (Interpreted
frame)
 -
org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=285, line=175 (Interpreted
frame)
 -
org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=64, line=128 (Interpreted
frame)
 -
org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=6, line=102 (Interpreted frame)
 -
org.apache.catalina.valves.AccessLogValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=24, line=563 (Interpreted
frame)
 -
org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=42, line=109 (Interpreted
frame)
 -
org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request,
org.apache.coyote.Response) @bci=157, line=263 (Interpreted frame)
 - org.apache.coyote.http11.Http11Processor.process(java.net.Socket)
@bci=432, line=844 (Interpreted frame)
 -
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(java.net.Socket)
@bci=82, line=584 (Interpreted frame)
 - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=447
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)





Yonik Seeley-2 wrote:
> 
> On Sun, Mar 1, 2009 at 10:32 AM, Jim Murphy <jim.murphy@pobox.com> wrote:
>> I should have said - tomcat is hosting 2 webapps a solr 1.3 master and
>> slave
>> - as separate web apps.
> 
> Given the the socket writes are blocked, it appears like whatever is
> supposed to be reading the other endpoint isn't doing it's job.
> 
> Are you using java-based replication?  Do you know if these sockets
> that are blocking are from client queries or from replication
> requests?  Splitting up the master and slave into separate JVMs might
> help shed some light on the situation.
> 
> -Yonik
> http://www.lucidimagination.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Tomcat-6-HTTP-Connector-Threads-all-blocked-tp22274107p22278035.html
Sent from the Solr - User mailing list archive at Nabble.com.


Mime
View raw message