tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier ...@ice-sa.com>
Subject Re: Apache Tomcat 6.0 Tomcat6 service stopped unexpectedly in windows server 2012
Date Thu, 07 Aug 2014 12:47:39 GMT
Alapan Sur wrote:
> Hi Andre,
> 
>  
> 
> Thanks for the quick reply.
> 
>  
> 
> I have verified the below requested memory related queries as suggested by
> you. The Initial memory pool and Maximum memory pool are empty and please
> find the attachment for the same.

This list usually strips attachments.
Either paste the info in-line (if it is text), or load the attachments somewhere else and

post a link.

> 
> 
> 
>  
> 
> We have checked the heap size with the command : java -XX:+PrintFlagsFinal
> -version | findstr /i "HeapSize PermSize ThreadStackSize"
> 
> 
> 
>  
> 
> Can you please guide how to increase the java memory pool heap size to 1GB ?
> 

I will, but please read this first, to understand how it works :

http://wiki.apache.org/tomcat/FAQ/Windows#Q11

It will save you time in the end, and will save us further questions which are already 
answered.
It will also explain why the numbers which you are getting with the above command, do not

represent the same settings as your Tomcat, running as a Windows Service.
As a Windows Service, the JVM which runs Tomcat uses the settings which you see in the GUI

program "tomcat6w.exe" (and which are stored in the Windows Registry).
This same program "tomcat6w.exe", is also the one which should appear as a little Tomcat 
icon on the task bar of your Windows server.
Thus :
- call up this GUI program
- navigate to the java tab
- set "Initial Memory pool" and "Maximum Memory pool" to, for example, 1024 MB
    (or more). Use the same value for both.
- click OK
- then restart the Tomcat Service

>  
> 
> Cheers,
> 
> Alapan
> 
> -----Original Message-----
> From: André Warnier [mailto:aw@ice-sa.com] 
> Sent: 07 August 2014 13:47
> To: Tomcat Users List
> Subject: Re: Apache Tomcat 6.0 Tomcat6 service stopped unexpectedly in
> windows server 2012
> 
>  
> 
> Alapan Sur wrote:
> 
>> Hi Team,
> 
> 
>>  
> 
> 
>> We have .Net application consuming tomcat solr3.5 services and 
> 
>> deployed on Windows Server 2012. Its is 64 bit machine with 32 GB RAM 
> 
>> and Tomcat veriosn is 6 and is installed at :"C:\Program Files 
> 
>> (x86)\Apache Software Foundation\Tomcat 6.0". During the peak load, 
> 
>> the solr sevice stops/unresponsive and it is happening very 
> 
>> frequently. We didnt fount any erro logs in windows event viewer but 
> 
>> in tomcat logs we got the following exception :
> 
> 
>> Aug 06, 2014 10:51:18 AM org.apache.solr.common.SolrException log SEVERE:
> 
>> ClientAbortException: java.net.SocketException: Software caused 
> 
>> connection
> 
>> abort: socket write error at
> 
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
> 
>> 369) at 
> 
>> org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339) at
> 
>> org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.jav
> 
>> a:392) at 
> 
>> org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381
> 
>> )
> 
>> at
> 
>> org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStr
> 
>> eam.ja
> 
>> va:89) at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) at 
> 
>> sun.nio.cs.StreamEncoder.implWrite(Unknown Source) at 
> 
>> sun.nio.cs.StreamEncoder.write(Unknown Source) at 
> 
>> java.io.OutputStreamWriter.write(Unknown Source) at
> 
>> org.apache.solr.common.util.FastWriter.write(FastWriter.java:55) at
> 
>> org.apache.solr.common.util.FastWriter.write(FastWriter.java:50) at
> 
>> org.apache.solr.response.XMLWriter.startTag(XMLWriter.java:269) at
> 
>> org.apache.solr.response.XMLWriter.writePrim(XMLWriter.java:780) at
> 
>> org.apache.solr.response.XMLWriter.writeInt(XMLWriter.java:691) at
> 
>> org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:565) at
> 
>> org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:621) 
> 
>> at
> 
>> org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:594) at
> 
>> org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:621) 
> 
>> at
> 
>> org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:594) at
> 
>> org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:621) 
> 
>> at
> 
>> org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:594) at
> 
>> org.apache.solr.response.XMLWriter.writeResponse(XMLWriter.java:132) 
> 
>> at
> 
>> org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.jav
> 
>> a:35)
> 
>> at
> 
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.
> 
>> java:343) at
> 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 
>> 265) at
> 
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> 
>> cation
> 
>> FilterChain.java:235) at
> 
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
> 
>> lterCh
> 
>> ain.java:206) at
> 
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> 
>> lve.ja
> 
>> va:233) at
> 
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> 
>> lve.ja
> 
>> va:191) at
> 
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> 
>> va:127
> 
>> ) at
> 
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
> 
>> va:103
> 
>> ) at
> 
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
> 
>> e.java
> 
>> :109) at
> 
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> 
>> :293)
> 
>> at
> 
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> 
>> 861)
> 
>> at
> 
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
> 
>> s(Http
> 
>> 11Protocol.java:606) at
> 
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489
> 
>> ) at java.lang.Thread.run(Unknown Source) Caused by:
> java.net.SocketException:
> 
>> Software caused connection abort: socket write error at 
> 
>> java.net.SocketOutputStream.socketWrite0(Native Method) at 
> 
>> java.net.SocketOutputStream.socketWrite(Unknown Source) at 
> 
>> java.net.SocketOutputStream.write(Unknown Source) at 
> 
>> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalO
> 
>> utputB
> 
>> uffer.java:761) at
> 
>> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:448) 
> 
>> at
> 
>> org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:363) at 
> 
>> org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer
> 
>> .doWri
> 
>> te(InternalOutputBuffer.java:785) at
> 
>> org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOu
> 
>> tputFi
> 
>> lter.java:126) at
> 
>> org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBu
> 
>> ffer.j
> 
>> ava:598) at org.apache.coyote.Response.doWrite(Response.java:560) at
> 
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
> 
>> 364) ... 36 more
> 
> 
>> Aug 06, 2014 10:51:29 AM org.apache.solr.common.SolrException log SEVERE:
> 
>> java.lang.OutOfMemoryError: Java heap space at
> 
>> org.apache.lucene.search.ExactPhraseScorer.(ExactPhraseScorer.java:38) 
> 
>> at
> 
>> org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.j
> 
>> ava:22
> 
>> 7) at
> 
>> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuer
> 
>> y.java
> 
>> :298) at
> 
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:577) 
> 
>> at
> 
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:364) 
> 
>> at 
> 
>> org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexS
> 
>> earche
> 
>> r.java:1408) at
> 
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:
> 
>> 1158) at
> 
>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java
> 
>> :362)
> 
>> at
> 
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponen
> 
>> t.java
> 
>> :378) at
> 
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear
> 
>> chHand
> 
>> ler.java:194) at
> 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
> 
>> java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372) 
> 
>> at
> 
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.
> 
>> java:3
> 
>> 56) at
> 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
> 
>> 252) at
> 
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
> 
>> cation
> 
>> FilterChain.java:235) at
> 
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
> 
>> lterCh
> 
>> ain.java:206) at
> 
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
> 
>> lve.ja
> 
>> va:233) at
> 
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
> 
>> lve.ja
> 
>> va:191) at
> 
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
> 
>> va:127
> 
>> ) at
> 
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
> 
>> va:103
> 
>> ) at
> 
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
> 
>> e.java
> 
>> :109) at
> 
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
> 
>> :293)
> 
>> at
> 
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> 
>> 861)
> 
>> at
> 
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
> 
>> s(Http
> 
>> 11Protocol.java:606) at
> 
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489
> 
>> ) at java.lang.Thread.run(Unknown Source)
> 
> 
>> Aug 07, 2014 11:53:10 AM org.apache.tomcat.util.net.JIoEndpoint
> 
>> createWorkerThread INFO: Maximum number of threads (200) created for 
> 
>> connector with address null and port 8080
> 
> 
>> Can anyone help on this error ?
> 
> 
>  
> 
> Maybe.
> 
> Let's start with the last one : that is an "INFO" message, not an error.  It
> tells you that this <Connector port="8080" ...>, which has a (probably
> default) configuration with the maxThreads attribute set to 200, has reached
> this value, and will not start any more additional threads to process new
> requests.  Which means that apparently, there are already 200 threads
> currently running and busy processing client requests at this time.
> 
> This could explain why the server is now "unresponsive" : after a while (as
> its "accept queue" fills up), it will stop accepting new client connections,
> as long as not at least one of these busy threads finishes what it is doing,
> and becomes available to process another client request.
> 
>  
> 
> Unfortunately the previous exeception (> Aug 06, 2014 10:51:29 AM
> org.apache.solr.common.SolrException log SEVERE:
> 
>> java.lang.OutOfMemoryError: Java heap space at..) kind of muddies the
> waters, because after an out-of-memory error, many strange and wonderful
> things can happen.
> 
> Including maybe 200 threads which would appear to be running and processing
> client requests, but in reality are not doing anything useful anymore.
> 
>  
> 
> In your description, you mention that the machine is 64-bit and has 32 GB
> RAM.
> 
> How much of that RAM is really available to the Java JVM which runs Tomcat ?
> And what are the settings for the Heap size of Java/Tomcat ?
> 
>  
> 
> To check this, run the "tomcat6w.exe" program which comes with Tomcat (in
> ...(tomcat_dir)/bin), and have a look at the java tab, which tells you what
> the "Initial/maximum memory pool" settings say.
> 
>  
> 
> You are running Solr/Lucene, which is a full-text search engine.  I am not a
> specialist of Solr/Lucene per se, but search engines in general tend to use
> a *lot* of memory to store intermediate search results, if the indexes being
> managed are large (aka the searched collections are large).
> 
> And, also in general, they tend to have a poor failover behaviour when they
> do not have enough memory to handle some search requests.
> 
> In the absence of any more information, and just making a somewhat
> uneducated guess, I would tend to say that anything less than 1 GB for the
> java "memory pool" (the Heap), is probably inadequate, unless this is just a
> small test system with a small collection of indexed items.
> 
>  
> 
> Note : if my memory serves right, Solr is the part which runs under Tomcat,
> and it is a front-end to Lucene, which is probably running in a separate
> Java process.
> 
> And I don't know who here is really using up the memory.  But your "out of
> Heap space" 
> 
> above is happening in Tomcat, so I'd concentrate on that first, and see
> whether getting rid of that problem improves the situation.
> 
>  
> 
>  
> 
> Ah, and by the way, tomcat 6 is rather old and outdated by now.  You may
> want to upgrade it to a more recent version (8, or at least 7).  That would
> get you better support in the future.
> 
>  
> 
>  
> 
>  
> 
> ---------------------------------------------------------------------
> 
> To unsubscribe, e-mail:  <mailto:users-unsubscribe@tomcat.apache.org>
> users-unsubscribe@tomcat.apache.org
> 
> For additional commands, e-mail:  <mailto:users-help@tomcat.apache.org>
> users-help@tomcat.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message