tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rainer Jung <rainer.j...@kippdata.de>
Subject Re: Apache/mod_jk serves random files from tomcat
Date Fri, 01 Aug 2008 08:24:31 GMT
dave.smith wrote:
> As I mentioned upgrading to mod_jk 1.2.26 was very easy.  Unfortunately,
> Tomcat is now crashing with "An unexpected error has been detected by
> HotSpot Virtual Machine."
> 
> #  SIGSEGV (0xb) at pc=0xb7aeaf7b, pid=19887, tid=2991246224
> #
> # Java VM: Java HotSpot(TM) Client VM (1.5.0_12-b04 mixed mode, sharing)
> # Problematic frame:
> # V  [libjvm.so+0x2ccf7b]
> #
> 
> There's a lot more.  Also, here are the jvm args:  
> jvm_args: -Djava.library.path=/usr/local/apr/lib -Xms512m -Xmx512m
> -XX:PermSize=256m -XX:MaxPermSize=256m
> -Djava.util.logging.manager=org.apache.juli.ClassLo
> aderLogManager
> -Djava.util.logging.config.file=/usr/share/tomcat5/conf/logging.properties
> -Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -Dcatalina.
> base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5
> -Djava.io.tmpdir=/usr/share/tomcat5/temp
> java_command: org.apache.catalina.startup.Bootstrap start
> 
> Do you believe these are related to the mod_jk upgrade?  It's happened twice
> (once on each server) since the upgrade that was made 2 weeks ago.
> 
> Please, let me know if you need anymore information.
> 
> Thanks,
> Dave

I don't know of any similar case. The full hot spot error file would be 
useful though. It e.g. contains a stack of the thread were the crash 
happened.

Regards,

Rainer

> Rainer Jung-3 wrote:
>> dave.smith schrieb:
>>> Yesterday, I upgraded our dev environment to mod_jk 1.2.26, which
>>> couldn't
>>> have been easier.  It will probably take me a couple of days before I can
>>> get this done in production, though.
>>>
>>> I terminate all HTTPS requests before they get to the web server, so from
>>> what you have described, it is probably safe to disable the APR
>>> connector. 
>>> How do I disable it, though?  I will look into disabling this after I
>>> have
>>> updated mod_jk in production.
>> Locate the tcnative shared object file (tcnative.so or tcnative-1.so) 
>> and renme it, so that the linker loader does not find it (e.g. add an 
>> underscore at the end of the file name).
>>
>> Then during the next startup, Tomcat should emit an info level log 
>> message telling you, that it couldn't find the lib.
>>
>>> Here's the full stack trace for that exception, displayed in my Tomcat
>>> logs:
>>>
>>> Jul 10, 2008 10:06:50 PM org.apache.catalina.connector.Request
>>> parseParameters
>>> WARNING: Exception thrown whilst processing POSTed parameters
>>> java.io.IOException: Socket read failed
>>>         at
>>> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
>>>         at
>>> org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1158)
>>>         at
>>> org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1090)
>>>         at
>>> org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1228)
>>>         at org.apache.coyote.Request.doRead(Request.java:419)
>>>         at
>>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
>>>         at
>>> org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
>>>         at
>>> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
>>>         at
>>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
>>>         at
>>> org.apache.catalina.connector.Request.readPostBody(Request.java:2400)
>>>         at
>>> org.apache.catalina.connector.Request.parseParameters(Request.java:2379)
>>>         at
>>> org.apache.catalina.connector.Request.getParameterNames(Request.java:1047)
>>>         at
>>> org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:369)
>>>         at
>>> org.apache.struts.util.RequestUtils.populate(RequestUtils.java:1225)
>>>         at
>>> org.apache.struts.action.RequestProcessor.processPopulate(RequestProcessor.java:821)
>>>         at
>>> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
>>>         at
>>> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
>>>         at
>>> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
>>>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>>>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>         at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>>         at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>         at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>>>         at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
>>>         at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>>         at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>>>         at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>>>         at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>>>         at
>>> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444)
>>>         at
>>> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:472)
>>>         at
>>> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
>>>         at java.lang.Thread.run(Thread.java:595)
>> OK, thanks. You are sure it is not 5.5.26?
>>
>> Regards,
>>
>> Rainer
>>
>>> Rainer Jung-3 wrote:
>>>> Hi David,
>>>>
>>>> dave.smith schrieb:
>>>>> Hi Rainer,
>>>>>
>>>>> Thanks a lot for the reply.
>>>>>
>>>>> I am using Tomcat 5.5.25 (rpm from jpackage.org).  CentOS Linux 2.6.18.
>>>>>
>>>>> httpd was compiled in prefork mode. The prefork settings are:
>>>>>
>>>>> StartServers       8
>>>>> MinSpareServers    5
>>>>> MaxSpareServers   20
>>>>> ServerLimit      256
>>>>> MaxClients       256
>>>>> MaxRequestsPerChild  4000
>>>>>
>>>>> I have setup JMeter to run against a test environment, but was unable
>>>>> to
>>>>> reproduce.  These random responses occur in production about once every
>>>>> week
>>>>> or so/more.  The problem will often (temporarily) correct itself, but
>>>>> sometimes I will need to restart httpd if the problem persists --
>>>>> restarting
>>>>> tomcat also works to temporarily correct the problem.
>>>>>
>>>>> The only thing strange that I see in my logs are in the
>>>>> test_client.log:
>>>>>
>>>>> WARNING: Exception thrown whilst processing POSTed parameters 
>>>>> java.io.IOException: Socket read failed
>>>>>         at
>>>>> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
>>>>>         ...
>>>> Thanks for the information. What is "test_client.log"? It looks like a 
>>>> Tomcat log file? Could you also post a larger part of the stack, or do 
>>>> you only get one line?
>>>>
>>>> Would you be able to do the following two things, maybe not both at the 
>>>> same time:
>>>>
>>>> - disable the apr connector (tcnative.so)
>>>> - upgrade jk to 1.2.26
>>>>
>>>> Concerning the apr connector: If you are using OpenSSL with apr and 
>>>> Tomcat or you have some similar reason you really need it, then don't 
>>>> switch. But if you use it without a very specific reason, disabling it 
>>>> for a week or two would help us isolate the problem.
>>>>
>>>> Concerning mod_jk upgrade: That should be very easy, apart from the 
>>>> following: if your httpd uses VirtualHost in the configuration, you have

>>>> to include your JkMount inside the VirtualHost, not in the global part, 
>>>> or you add JkMountCopy On to the VirtualHost.
>>>>
>>>> Regards,
>>>>
>>>> Rainer
>>>>
>>>>> Rainer Jung-3 wrote:
>>>>>> dave.smith schrieb:
>>>>>>>> Wow. That's weird. Is Tomcat serving the file, or is httpd
serving
>>>>>>>> it?
>>>>>>> Not too weird.  I am experiencing the same thing with Tomcat
5.5 and
>>>>>>> mod_jk
>>>>>>> 1.2.23.  I have Tomcat serving everything.
>>>>>>>
>>>>>>> I am also using a load balancer that sends an OPTION every 2
seconds
>>>>>>> to
>>>>>>> each
>>>>>>> web server to make sure that the server is alive.
>>>>>>>
>>>>>>> This intermittent random response issue is really killing me.
>>>>>> Could you please also add some info:
>>>>>>
>>>>>> Tomcat version?
>>>>>>
>>>>>> And from my previous mail:
>>>>>>
>>>>>> What's you platform and which httpd MPM (prefork orworker or something

>>>>>> else) do you use? For some platforms (e.g. AIX) the detection of

>>>>>> multi-threading in httpd during mpod_jk build-time was broken.
>>>>>> Starting 
>>>>>> with 1.2.24 we build always including multi-thread support unless

>>>>>> explicitely stated via a configure option. If you 1.2.23 build is
not 
>>>>>> thread safe, but your httpd uses threads (like with worker mpm),
then 
>>>>>> such trouble is possible, although more likely you would see crashes

>>>>>> etc. For most platforms like Linux and Solaris the threading detection

>>>>>> was OK already before 1.2.24.
>>>>>>
>>>>>> Another possible (but not very likely) cause could be bug 44494 of

>>>>>> Tomcat 6.0.16/5.5.26 which under certain circumstances could leave
>>>>>> data 
>>>>>> in the request object after request handling completed. You could
try 
>>>>>> either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be

>>>>>> expected 6.0.17/5.5.27.
>>>>>>
>>>>>> I would also add the access log on the Tomcat side. If you find the
>>>>>> same 
>>>>>> phenomenon there, then it's unlikely, that httpd/mod_jk are
>>>>>> responsible 
>>>>>> and the reason should be inside Tomcat or the webapp.
>>>>>>
>>>>>> Can you reproduce the problem on a test system?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message