tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirill Ilyukhin <>
Subject Re: Webapp reload corrupts Tomcat
Date Mon, 25 Feb 2013 02:27:38 GMT

On 25.02.2013 2:10, Daniel Mikusa wrote:
> On Feb 24, 2013, at 11:48 AM, Kirill Ilyukhin wrote:
>> On Feb 22, 2013, at 22:32 , Daniel Mikusa <> wrote:
>>> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>>>> Hi!
>>>> I have 5 servers with the following configuration:
>>>>   Windows Server 2008 R2 64-bit,
>>>>   Oracle JDK 1.7.0_11-b21 64-bit,
>>>>   Tomcat 7.0.35 running as a service.
>>>> A few webapps are running perfectly well under the Tomcat until one of
>>>> them is reloaded.
>>>> Reload of a webapp (either by pressing [reload] on manager page or by
>>>> changing web.xml file) makes Tomcat extremely unstable.
>>>> The first (pre-production) server sporadically fails with absolutely
>>>> no traces in log files or Windows event logs:
>>> These three symptoms can often mean different things.  For example, if it's not
accepting connections then you can't really get a 404 error.  Can you be more specific about
when you see them and the status of your Tomcat server when you see them.
>>> Some troubleshooting thoughts...
>>>> does not accept connections,
>>> 1.) Is the process still running?  Did the JVM crash?  Is there a crash file?
>>> 2.) Is the socket still listening?  Have you changed any firewall rules?
>>> 3.) Try "telnet <port>".  What happens?
>>> 4.) Try connecting to Tomcat with jconsole or jvisualvm?  Can you?  If so look
at the connector mbean. What is its state?
>>> 5.) Take some thread dumps.
>>>> responds with HTTP 404 or
>>> Assuming the URL that you entered is valid, this sounds like your application
may have failed to reload.  Can you turn up the log level in your application to see what
it is doing?
>>>> does not respond at all.
>>> Assuming that you mean it accepts your connection and request, but never processes
and returns a response.  In this case, take some thread dumps after you have sent the response.
 That should show you what is going on inside the JVM.
>> Here is more details.
>> All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
>> A webapp is serving real-time data with updates. Open the page in a browser, it downloads
an HTML (a static file), then a bunch of CSS and Javascripts (also static files), and starts
fetching the data and updates (from a servlet) - this makes an indefinite sequence of HTTP
>> When I open the page after the webapp is reloaded, some of this requests (including
the static files) fail with either 404 code or no response or not accepted connection. Requests
are being failed quite randomly. If I reload the page, also a lot of fails, but slightly different
- e.g. I might get the file which was 404 earlier. The data requests also fail randomly, some
of them are successful, which means that the webapp is loaded.
>> So, answering to the questions above: the process is definitely running, socket is
definitely listening, telneting Tomcat gives random results, JConsole shows nothing uncommon.
> Can you attach the access log that shows these requests?
I do not have the access logs at my disposal to attach them. From what I 
remember, the access logs were clean - no 404 codes, every request is 
responded. In other words if you look from the server side (logs, 
JConsole, Windows logs) , everything is fine.
> Also, a couple more follow up questions…
> 1.) When this occurs does it only occur for the application that you reload?  or does
it happen for all the apps that you have deployed to the Tomcat server?
It happens to all the webapps on the server, even "pre-installed" ones 
(manager and ROOT).
> 2.) Is this application specific, in other words does this only occur when you reload
one particular application?  or does reloading any application on the Tomcat server cause
the problem?
It is random. One of the application, when reloaded, causes the problem 
almost for sure. Others can also cause the problem, but less frequently.
> 3.) Can you replicate this problem in a testing environment?
No, I can not.
>>>> Other 4 (production) servers also become unstable but in a different
>>>> manner - a lot of HTTP 500 responses (logs are also clean).
>>> It would be very odd to see nothing in the logs after a 500 error.  Have you
looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"?  That file should
contain a stack trace after a 500 error.  Also, what do the see in your browser when you get
the 500 error?  The default error page should print the stack trace for the 500 error.
>> That is what I am talking about - quite odd behaviour. Nothing in log files, no body
with HTTP 500 response in a browser.
>>>> Plus on
>>>> some of these servers I have a lot (less than HTTP 500 responses
>>>> though) of NPEs:
>>>> -------------
>>>> SEVERE: Exception Processing /path/here
>>>> java.lang.NullPointerException
>>>> 	at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(
>>>> 	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(
>>>> 	at org.apache.catalina.valves.RequestFilterValve.process(
>>>> 	at org.apache.catalina.valves.RemoteAddrValve.invoke(
>>>> 	at org.apache.catalina.core.StandardHostValve.invoke(
>>>> 	at org.apache.catalina.valves.ErrorReportValve.invoke(
>>>> 	at org.apache.catalina.valves.AccessLogValve.invoke(
>>>> 	at org.apache.catalina.core.StandardEngineValve.invoke(
>>>> 	at org.apache.catalina.connector.CoyoteAdapter.service(
>>>> 	at org.apache.coyote.http11.AbstractHttp11Processor.process(
>>>> 	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(
>>>> 	at$
>>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> 	at java.util.concurrent.ThreadPoolExecutor$
>>>> 	at
>>>> -------------
>>>> These exception occurs for any webapp, not only the one being reloaded.
>>> What's the specific version of Tomcat where you saw this error occur?  Is it
7.0.37? or was this on an older version?
>> This was on 7.0.35
>>>> All the above happens irregularly, quite often on the pre-production
>>>> server, only once on production servers. Tomcat service restart fixes
>>>> the problem. Updating Tomcat to 7.0.37 did not help.
>>>> I cannot reproduce this on any of developers hosts with the same
>>>> versions of JDK and Tomcat.
>>>> Any ideas?
>>> 1.) An easy temporary fix would be to not reload the application in production.
 If you need to redeploy or reload the application, stop the server, undeploy, redeploy and
start your server.
>>> 2.) Include your Tomcat configuration, minus comments.
>> Please see below:
>> <Server port="8005" shutdown="SHUTDOWN">
>>   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"
>>   <Listener className="org.apache.catalina.core.JasperListener" />
>>   <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>   <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>   <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>>   <GlobalNamingResources>
>>     <Resource name="UserDatabase" auth="Container"
>>               type="org.apache.catalina.UserDatabase"
>>               description="User database that can be updated and saved"
>>               factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>               pathname="conf/tomcat-users.xml" />
>>     <!-- A couple of custom resources here -->
>>   </GlobalNamingResources>
>>   <Service name="Catalina">
>>     <Connector port="80" protocol="HTTP/1.1"
>>                asyncTimeout="20000"
>>                URIEncoding="utf-8"
>>                acceptorThreadCount="2"
>>                compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
>>                compression="on"
> You might want to disable compression and see if that makes a difference.  I can't say
that this is specifically causing the problem, but it would allow you to rule it out as a
Thanks, I will try disabling compression, but, as It might not help, 
that is not enough for me. Servers are located in a data centre with 
limited access and I should have some strong ideas of how to fix the 
problem (or detect its cause) when I go there next time.

As I have said, everything worked fine with JDK 1.6. Can this be because 
of some unexpected behaviour of Oracle JDK 1.7 classloader for win64? 
Can it be because class files of the most of the apps were build with 
JDK 1.6 and now running on Java 7? Can it be because some apps may leak 
threads and ThreadLocals and Java 6 JVM could handle it but Java 7 JVM 


> Dan
>>                connectionTimeout="20000"
>>                maxThreads="1024"
>>                processorCache="512"
>>                redirectPort="8443" />
>>     <Engine name="Catalina" defaultHost="localhost">
>>       <Realm className="org.apache.catalina.realm.LockOutRealm">
>>         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>                resourceName="UserDatabase"/>
>>       </Realm>
>>       <Host name="localhost"  appBase="webapps"
>>             unpackWARs="true" autoDeploy="true">
>>         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
>>                prefix="localhost_access_log." suffix=".txt"
>>                pattern="%h %l %u %t &quot;%r&quot; %s %b %Dms" resolveHosts="false"/>
>>       </Host>
>>     </Engine>
>>   </Service>
>> </Server>
>> Best regards,
>> Kirill
>>> 3.) See troubleshooting steps above.
>>> Dan
>>>> Thanks a lot in advance,
>>>> Kirill
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail:
>>>> For additional commands, e-mail:
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message