tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Mikusa <dmik...@vmware.com>
Subject Re: Webapp reload corrupts Tomcat
Date Mon, 25 Feb 2013 13:58:04 GMT
On Feb 24, 2013, at 9:27 PM, Kirill Ilyukhin wrote:

> 
> On 25.02.2013 2:10, Daniel Mikusa wrote:
>> On Feb 24, 2013, at 11:48 AM, Kirill Ilyukhin wrote:
>> 
>>> On Feb 22, 2013, at 22:32 , Daniel Mikusa <dmikusa@vmware.com> wrote:
>>> 
>>>> On Feb 22, 2013, at 12:09 AM, Kirill Ilyukhin wrote:
>>>> 
>>>>> Hi!
>>>>> 
>>>>> I have 5 servers with the following configuration:
>>>>>  Windows Server 2008 R2 64-bit,
>>>>>  Oracle JDK 1.7.0_11-b21 64-bit,
>>>>>  Tomcat 7.0.35 running as a service.
>>>>> A few webapps are running perfectly well under the Tomcat until one of
>>>>> them is reloaded.
>>>>> 
>>>>> Reload of a webapp (either by pressing [reload] on manager page or by
>>>>> changing web.xml file) makes Tomcat extremely unstable.
>>>>> The first (pre-production) server sporadically fails with absolutely
>>>>> no traces in log files or Windows event logs:
>>>> These three symptoms can often mean different things.  For example, if it's
not accepting connections then you can't really get a 404 error.  Can you be more specific
about when you see them and the status of your Tomcat server when you see them.
>>>> 
>>>> Some troubleshooting thoughts...
>>>> 
>>>>> does not accept connections,
>>>> 1.) Is the process still running?  Did the JVM crash?  Is there a crash file?
 hs_err_podXXXX.log?
>>>> 2.) Is the socket still listening?  Have you changed any firewall rules?
>>>> 3.) Try "telnet <port>".  What happens?
>>>> 4.) Try connecting to Tomcat with jconsole or jvisualvm?  Can you?  If so
look at the connector mbean. What is its state?
>>>> 5.) Take some thread dumps.
>>>> 
>>>>> responds with HTTP 404 or
>>>> Assuming the URL that you entered is valid, this sounds like your application
may have failed to reload.  Can you turn up the log level in your application to see what
it is doing?
>>>> 
>>>>> does not respond at all.
>>>> Assuming that you mean it accepts your connection and request, but never
processes and returns a response.  In this case, take some thread dumps after you have sent
the response.  That should show you what is going on inside the JVM.
>>> Here is more details.
>>> All these was perfectly working on JDK 1.6 and Tomcat 7 a few builds earlier.
>>> A webapp is serving real-time data with updates. Open the page in a browser,
it downloads an HTML (a static file), then a bunch of CSS and Javascripts (also static files),
and starts fetching the data and updates (from a servlet) - this makes an indefinite sequence
of HTTP requests.
>>> When I open the page after the webapp is reloaded, some of this requests (including
the static files) fail with either 404 code or no response or not accepted connection. Requests
are being failed quite randomly. If I reload the page, also a lot of fails, but slightly different
- e.g. I might get the file which was 404 earlier. The data requests also fail randomly, some
of them are successful, which means that the webapp is loaded.
>>> So, answering to the questions above: the process is definitely running, socket
is definitely listening, telneting Tomcat gives random results, JConsole shows nothing uncommon.
>> Can you attach the access log that shows these requests?
> I do not have the access logs at my disposal to attach them. From what I remember, the
access logs were clean - no 404 codes, every request is responded. In other words if you look
from the server side (logs, JConsole, Windows logs) , everything is fine.
>>  
>> Also, a couple more follow up questions…
>> 
>> 1.) When this occurs does it only occur for the application that you reload?  or
does it happen for all the apps that you have deployed to the Tomcat server?
> It happens to all the webapps on the server, even "pre-installed" ones (manager and ROOT).
>> 2.) Is this application specific, in other words does this only occur when you reload
one particular application?  or does reloading any application on the Tomcat server cause
the problem?
> It is random. One of the application, when reloaded, causes the problem almost for sure.
Others can also cause the problem, but less frequently.
>> 
>> 3.) Can you replicate this problem in a testing environment?
> No, I can not.
>> 
>>>>> Other 4 (production) servers also become unstable but in a different
>>>>> manner - a lot of HTTP 500 responses (logs are also clean).
>>>> It would be very odd to see nothing in the logs after a 500 error.  Have
you looked at all of the log files, in particular "localhost-YYYY-MM-DD.log"?  That file should
contain a stack trace after a 500 error.  Also, what do the see in your browser when you get
the 500 error?  The default error page should print the stack trace for the 500 error.
>>> That is what I am talking about - quite odd behaviour. Nothing in log files,
no body with HTTP 500 response in a browser.
>>> 
>>>>> Plus on
>>>>> some of these servers I have a lot (less than HTTP 500 responses
>>>>> though) of NPEs:
>>>>> -------------
>>>>> SEVERE: Exception Processing /path/here
>>>>> java.lang.NullPointerException
>>>>> 	at org.apache.catalina.core.StandardWrapper.servletSecurityAnnotationScan(StandardWrapper.java:1216)
>>>>> 	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)
>>>>> 	at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:305)
>>>>> 	at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:83)
>>>>> 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>> 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>>> 	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>>> 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>> 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>>> 	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>>> 	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>>> 	at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1822)
>>>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>> 	at java.lang.Thread.run(Thread.java:722)
>>>>> -------------
>>>>> These exception occurs for any webapp, not only the one being reloaded.
>>>> What's the specific version of Tomcat where you saw this error occur?  Is
it 7.0.37? or was this on an older version?
>>> This was on 7.0.35
>>> 
>>>>> All the above happens irregularly, quite often on the pre-production
>>>>> server, only once on production servers. Tomcat service restart fixes
>>>>> the problem. Updating Tomcat to 7.0.37 did not help.
>>>>> I cannot reproduce this on any of developers hosts with the same
>>>>> versions of JDK and Tomcat.
>>>>> 
>>>>> Any ideas?
>>>> 1.) An easy temporary fix would be to not reload the application in production.
 If you need to redeploy or reload the application, stop the server, undeploy, redeploy and
start your server.
>>>> 2.) Include your Tomcat configuration, minus comments.
>>> Please see below:
>>> <Server port="8005" shutdown="SHUTDOWN">
>>>  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"
/>
>>>  <Listener className="org.apache.catalina.core.JasperListener" />
>>>  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
/>
>>>  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>
>>>  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
/>
>>>  <GlobalNamingResources>
>>>    <Resource name="UserDatabase" auth="Container"
>>>              type="org.apache.catalina.UserDatabase"
>>>              description="User database that can be updated and saved"
>>>              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>              pathname="conf/tomcat-users.xml" />
>>>    <!-- A couple of custom resources here -->
>>>  </GlobalNamingResources>
>>>  <Service name="Catalina">
>>>    <Connector port="80" protocol="HTTP/1.1"
>>>               asyncTimeout="20000"
>>>               URIEncoding="utf-8"
>>>               acceptorThreadCount="2"
>>>               compressableMimeType="text/html,text/xml,text/plain,text/x-json,application/javascript,application/json,text/css"
>>>               compression="on"
>> You might want to disable compression and see if that makes a difference.  I can't
say that this is specifically causing the problem, but it would allow you to rule it out as
a possibility.
> Thanks, I will try disabling compression, but, as It might not help, that is not enough
for me. Servers are located in a data centre with limited access and I should have some strong
ideas of how to fix the problem (or detect its cause) when I go there next time.

Some other thoughts for troubleshooting this…

1.) Take a heap dump and a couple thread dumps.  Perform the reload.  Confirm the problem.
 Take another heap dump and some more thread dumps.  Compare to see what has changed and if
you see anything odd.

2.) Setup a packet capture like Wireshark or tcpdump.  Capture some packets prior to the reload,
capture some after the reload.  Compare and look for a difference in the requests and responses.

3.) If you haven't already, update to the latest Java 7.  You could be hitting a bug.

> As I have said, everything worked fine with JDK 1.6. Can this be because of some unexpected
behaviour of Oracle JDK 1.7 classloader for win64? Can it be because class files of the most
of the apps were build with JDK 1.6 and now running on Java 7? Can it be because some apps
may leak threads and ThreadLocals and Java 6 JVM could handle it but Java 7 JVM cannot?

Hard to say exactly.  What I can say is that there are tons of people running Tomcat on Java
7 without issue, so I would suggest focusing on anything that might be different or non-standard
about the environment where you are seeing the issue.  For example, can you replicate the
problem with fresh Tomcat installation (i.e. download, unzip, deploy your app)?  Can you replicate
the problem without deploying your apps (i.e. with just standard TC apps deployed)?  

Hope that helps

Dan


> 
> Thanks,
> Kirill
> 
>> 
>> Dan
>> 
>> 
>>>               connectionTimeout="20000"
>>>               maxThreads="1024"
>>>               processorCache="512"
>>>               redirectPort="8443" />
>>>    <Engine name="Catalina" defaultHost="localhost">
>>>      <Realm className="org.apache.catalina.realm.LockOutRealm">
>>>        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>>               resourceName="UserDatabase"/>
>>>      </Realm>
>>>      <Host name="localhost"  appBase="webapps"
>>>            unpackWARs="true" autoDeploy="true">
>>>        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
>>>               prefix="localhost_access_log." suffix=".txt"
>>>               pattern="%h %l %u %t &quot;%r&quot; %s %b %Dms" resolveHosts="false"/>
>>>      </Host>
>>>    </Engine>
>>>  </Service>
>>> </Server>
>>> 
>>> 
>>> Best regards,
>>> Kirill
>>> 
>>> 
>>>> 3.) See troubleshooting steps above.
>>>> 
>>>> Dan
>>>> 
>>>>> 
>>>>> Thanks a lot in advance,
>>>>> Kirill
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message