lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dušan Omerčević (JIRA) <j...@apache.org>
Subject [jira] [Commented] (SOLR-3544) Under heavy load json response is cut at some arbitrary position
Date Fri, 15 Jun 2012 20:35:43 GMT

    [ https://issues.apache.org/jira/browse/SOLR-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295913#comment-13295913
] 

Dušan Omerčević commented on SOLR-3544:
---------------------------------------

Ad 1) We're using Tomcat.

Ad 2) We fetch up to 40K documents with each document being a few KB in size. Total response
is some 100MB in size. The cut off really happened at arbitrary position, from a few KB into
the response up to several MB into the response.

Ad 3) I've checked it thoroughly and I couldn't find any pattern. It really seems that the
problem is independent of the data.

Ad 4) The full message is:
2012-06-11 12:23:34 ERROR: ClientAbortException:  java.net.SocketException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:370)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:323)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:396)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:385)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
        at org.apache.solr.common.util.FastWriter.flush(FastWriter.java:113)
        at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:344)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:873)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:750)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:432)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:347)
        at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:773)
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:127)
        at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:583)
        at org.apache.coyote.Response.doWrite(Response.java:560)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:365)
        ... 25 more

2012-06-11 12:23:34 ERROR: Servlet.service() for servlet default threw exception
java.lang.IllegalStateException
        at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:405)
        at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:380)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:283)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:873)
        at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
        at java.lang.Thread.run(Thread.java:662)

But I believe that this is just a secondary effect of client closing the connection. It seems
that the client is closing the connection because he doesn't have anything left to read. And
it seems that it doesn't have anything to read because server abruptly stops sending data.
In catalina.out there were no other error messages that would indicate why the server stopped
sending data.


I hope this helps at least a bit. I didn't post this bug report because I expected it to be
resolved (it's quite weird indeed), but mostly as reference point if somebody else is experiencing
similar problems. But if it would get resolved that would be even better, of course :)

Dušan
                
> Under heavy load json response is cut at some arbitrary position
> ----------------------------------------------------------------
>
>                 Key: SOLR-3544
>                 URL: https://issues.apache.org/jira/browse/SOLR-3544
>             Project: Solr
>          Issue Type: Bug
>          Components: search
>    Affects Versions: 3.1
>         Environment: Linux version 2.6.32-5-amd64 (Debian 2.6.32-38) (ben@decadent.org.uk)
(gcc version 4.3.5 (Debian 4.3.5-4) )
>            Reporter: Dušan Omerčević
>
> We query solr for 30K documents using json as the response format. Normally this works
perfectly fine. But when the machine comes under heavy load (all cores utilized) the response
got interrupted at arbitrary position. We circumvented the problem by switching to xml response
format.
> I've written the full description here: http://restreaming.wordpress.com/2012/06/14/the-curious-case-of-solr-malfunction/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message