Linux OOM killer?
http://lwn.net/Articles/317814/
p
On 07/07/2011 09:35, martin@alt-v.co.uk wrote:
> Yeh, I've worked out what was happening, seems that tomcat was being
> killed by the system running out of memory, which was happening without=
> any log messages on the system at all, which was rather unhelpful and
> left me completely stumped for a while. However adding extra swap spac=
e
> and memory seem to fix the problem.
>=20
> Thanks anyway for the reply
>=20
> Quoting Rainer Jung <rainer.jung@kippdata.de>:
>=20
>> It seems it is not a mod_jk problem. mod_jk never shuts down Tomcat, b=
ut
>> obviously your Tomcat shuts down during processing the request.
>>
>> I would
>>
>> - update Tomcat *and* the service wrapper (jdvc) to the most recent
>> patch update (e.g. Tomcat 5.5.33).
>>
>> - run Tomcat once without the service wrapper to decide, whether the
>> problem comes from the wrapper or from Tomcat respectively your web
>> application.
>>
>> Regards,
>>
>> Rainer
>>
>> On 04.07.2011 18:27, martin@alt-v.co.uk wrote:
>>> Hi
>>> I have a problem when uploading files to tomcat through mod_jk. When=
>>> uploading smaller files (<100kb) everything seems to work fine, but w=
hen
>>> uploading files of about 1.5Mb it seems as if the server is restartin=
g
>>> and causing a 503 error. I've tried modifying timeouts in both apach=
e
>>> and tomcat, still i get this output on the catalina.err file: (sorry =
for
>>> long log files)
>>>
>>>
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:31 7211 jsvc.exec error: Shutdown or reload already
>>> scheduled
>>> 04/07/2011 15:19:32 7210 jsvc.exec error: Service killed by signal 9
>>> Jul 4, 2011 3:19:35 PM org.apache.catalina.core.AprLifecycleListener
>>> init
>>> INFO: The APR based Apache Tomcat Native library which allows optimal=
>>> performance in production environments was not found on the
>>> java.library.path:
>>> /usr/local/jdk/jre/lib/i386/client:/usr/local/jdk/jre/lib/i386:/usr/j=
ava/packages/lib/i386:/lib:/usr/lib
>>>
>>>
>>> Jul 4, 2011 3:19:35 PM org.apache.coyote.http11.Http11BaseProtocol in=
it
>>> INFO: Initializing Coyote HTTP/1.1 on http-8080
>>> Jul 4, 2011 3:19:35 PM org.apache.catalina.startup.Catalina load
>>> INFO: Initialization processed in 426 ms
>>> Jul 4, 2011 3:19:35 PM org.apache.catalina.core.StandardService start=
>>> INFO: Starting service Catalina
>>> Jul 4, 2011 3:19:35 PM org.apache.catalina.core.StandardEngine start
>>> INFO: Starting Servlet Engine: Apache Tomcat/5.5.33
>>> Jul 4, 2011 3:19:35 PM org.apache.catalina.core.StandardHost start
>>> INFO: XML validation disabled
>>> Jul 4, 2011 3:19:35 PM org.apache.catalina.core.StandardHost start
>>> INFO: XML validation disabled
>>> Jul 4, 2011 3:19:36 PM org.apache.catalina.core.StandardHost start
>>> INFO: XML validation disabled
>>> Jul 4, 2011 3:19:36 PM org.apache.catalina.core.StandardHost start
>>> INFO: XML validation disabled
>>> Jul 4, 2011 3:19:36 PM org.apache.catalina.core.StandardHost start
>>> INFO: XML validation disabled
>>> Jul 4, 2011 3:19:37 PM org.apache.coyote.http11.Http11BaseProtocol st=
art
>>> INFO: Starting Coyote HTTP/1.1 on http-8080
>>> Jul 4, 2011 3:19:37 PM org.apache.jk.common.ChannelSocket init
>>> INFO: JK: ajp13 listening on /0.0.0.0:8009
>>> Jul 4, 2011 3:19:37 PM org.apache.jk.server.JkMain start
>>> INFO: Jk running ID=3D0 time=3D0/246 config=3Dnull
>>> Jul 4, 2011 3:19:37 PM org.apache.catalina.storeconfig.StoreLoader lo=
ad
>>> INFO: Find registry server-registry.xml at classpath resource
>>> Jul 4, 2011 3:19:37 PM org.apache.catalina.startup.Catalina start
>>> INFO: Server startup in 2419 ms
>>> ----------
>>>
>>> The error from the apache mod_jk.log file is:
>>>
>>> [Mon Jul 04 15:18:09 2011] [7413:3080754960] [info] init_jk::mod_jk.c=
>>> (3189): mod_jk/1.2.30 initialized
>>> [Mon Jul 04 15:19:32 2011] [7460:3080754960] [info]
>>> ajp_connection_tcp_get_message::jk_ajp_common.c (1223): (ajp13) can't=
>>> receive the response header message from tomcat, tomcat (127.0.0.1:80=
09)
>>> has forced a connection close for socket 2
>>> 7
>>> [Mon Jul 04 15:19:32 2011] [7460:3080754960] [error]
>>> ajp_get_reply::jk_ajp_common.c (2055): (ajp13) Tomcat is down or refu=
sed
>>> connection. No response has been sent to the client (yet)
>>> [Mon Jul 04 15:19:32 2011] [7460:3080754960] [error]
>>> ajp_service::jk_ajp_common.c (2533): (ajp13) sending request to tomca=
t
>>> failed (unrecoverable), (attempt=3D1)
>>> [Mon Jul 04 15:19:32 2011] ajp13 barcodes3.alt-v.co.uk 27.361459
>>> [Mon Jul 04 15:19:32 2011] [7460:3080754960] [info] jk_handler::mod_j=
k.c
>>> (2618): Service error=3D0 for worker=3Dajp13
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [info]
>>> jk_open_socket::jk_connect.c (622): connect to 127.0.0.1:8009 failed
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [info]
>>> ajp_connect_to_endpoint::jk_ajp_common.c (959): Failed opening socket=
to
>>> (127.0.0.1:8009) (errno=3D111)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [error]
>>> ajp_send_request::jk_ajp_common.c (1585): (ajp13) connecting to backe=
nd
>>> failed. Tomcat is probably not started or is listening on the wrong p=
ort
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [info]
>>> ajp_service::jk_ajp_common.c (2540): (ajp13) sending request to tomca=
t
>>> failed (recoverable), because of error during request sending
>>> (attempt=3D1)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [info]
>>> jk_open_socket::jk_connect.c (622): connect to 127.0.0.1:8009 failed
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [info]
>>> ajp_connect_to_endpoint::jk_ajp_common.c (959): Failed opening socket=
to
>>> (127.0.0.1:8009) (errno=3D111)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [error]
>>> ajp_send_request::jk_ajp_common.c (1585): (ajp13) connecting to backe=
nd
>>> failed. Tomcat is probably not started or is listening on the wrong p=
ort
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [info]
>>> ajp_service::jk_ajp_common.c (2540): (ajp13) sending request to tomca=
t
>>> failed (recoverable), because of error during request sending
>>> (attempt=3D2)
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [error]
>>> ajp_service::jk_ajp_common.c (2559): (ajp13) connecting to tomcat
>>> failed.
>>> [Mon Jul 04 15:19:33 2011] ajp13 barcodes3.alt-v.co.uk 0.116266
>>> [Mon Jul 04 15:19:33 2011] [7464:3080754960] [info] jk_handler::mod_j=
k.c
>>> (2618): Service error=3D-3 for worker=3Dajp13
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [info]
>>> jk_open_socket::jk_connect.c (622): connect to 127.0.0.1:8009 failed
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [info]
>>> ajp_connect_to_endpoint::jk_ajp_common.c (959): Failed opening socket=
to
>>> (127.0.0.1:8009) (errno=3D111)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [error]
>>> ajp_send_request::jk_ajp_common.c (1585): (ajp13) connecting to backe=
nd
>>> failed. Tomcat is probably not started or is listening on the wrong p=
ort
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [info]
>>> ajp_service::jk_ajp_common.c (2540): (ajp13) sending request to tomca=
t
>>> failed (recoverable), because of error during request sending
>>> (attempt=3D1)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [info]
>>> jk_open_socket::jk_connect.c (622): connect to 127.0.0.1:8009 failed
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [info]
>>> ajp_connect_to_endpoint::jk_ajp_common.c (959): Failed opening socket=
to
>>> (127.0.0.1:8009) (errno=3D111)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [error]
>>> ajp_send_request::jk_ajp_common.c (1585): (ajp13) connecting to backe=
nd
>>> failed. Tomcat is probably not started or is listening on the wrong p=
ort
>>> (errno=3D111)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [info]
>>> ajp_service::jk_ajp_common.c (2540): (ajp13) sending request to tomca=
t
>>> failed (recoverable), because of error during request sending
>>> (attempt=3D2)
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [error]
>>> ajp_service::jk_ajp_common.c (2559): (ajp13) connecting to tomcat
>>> failed.
>>> [Mon Jul 04 15:19:34 2011] ajp13 barcodes3.alt-v.co.uk 0.100502
>>> [Mon Jul 04 15:19:34 2011] [7467:3080754960] [info] jk_handler::mod_j=
k.c
>>> (2618): Service error=3D-3 for worker=3Dajp13
>>>
>>>
>>> ------
>>>
>>>
>>>
>>>
>>> So I think its restarting the server for some reason. As far as I ca=
n
>>> tell I'm not getting any errors on the startup of the server. Is the=
re
>>> any way to find out more about whats happening and maybe fix this
>>> problem. My workers.conf is:
>>>
>>> worker.ajp13.port=3D8009
>>> worker.ajp13.host=3Dlocalhost
>>> worker.ajp13.type=3Dajp13
>>> #
>>> # Specifies the load balance factor when used with
>>> # a load balancing worker.
>>> # Note:
>>> # ----> lbfactor must be > 0
>>> # ----> Low lbfactor means less work done by the worker.
>>>
>>> worker.ajp13.lbfactor=3D1
>>>
>>> #
>>> # Specify the size of the open connection pool.
>>> #worker.ajp13.connection_pool_size
>>> connection_pool_timeout=3D600
>>> connectionTimeout=3D600000
>>> #
>>>
>>>
>>> The connector for the server is:
>>>
>>> <Connector port=3D"8009"
>>> maxProcessors=3D"50" enableLookups=3D"false"
>>> disableUploadTimeout=3D"true" protocol=3D"AJP/1.3"
>>> connectionTimeout=3D"180000" maxSavePostSize=3D"-1" maxPostSize=3D"0"=
/>
>>>
>>>
>>> I would post the rest of the file if its needed, but i think thats th=
e
>>> most important part.
>>>
>>> The jk.conf from apache is:
>>>
>>> JkWorkersFile /usr/local/jakarta/tomcat/conf/workers.properties
>>> JkLogFile /usr/local/apache/logs/mod_jk.log
>>> JkLogLevel info
>>> JkWatchdogInterval 600
>>> JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
>>> JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
>>> JkRequestLogFormat "%w %V %T"
>>>
>>> I'm using tomcat 5.5, it would be difficult for me to upgrade. Can
>>> anyone point out anything that I might be doing wrong, or if you need=
>>> any more info let me know
>>>
>>> Thanks
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>=20
>=20
>=20
>=20
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>=20
|