httpd-bugs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bugzi...@apache.org
Subject [Bug 58511] New: apache-tomcat - APR does not understand this error code
Date Wed, 21 Oct 2015 08:34:53 GMT
https://bz.apache.org/bugzilla/show_bug.cgi?id=58511

            Bug ID: 58511
           Summary: apache-tomcat - APR does not understand this error
                    code
           Product: Apache httpd-2
           Version: 2.2.27
          Hardware: PC
            Status: NEW
          Severity: normal
          Priority: P2
         Component: mod_proxy_ajp
          Assignee: bugs@httpd.apache.org
          Reporter: nihil.somani@gmail.com

We are on production with Apache/2.2.27 and Tomcat Version 7.0.55 on
linux(2.6.32-431.37.1.el6.x86_64) with 6 core. what we're seeing is below error
on webserver which has no pattern as to when it occurs.

(104)Connection reset by peer: ajp_ilink_receive() can't receive header
ajp_read_header: ajp_ilink_receive failed
(120006)APR does not understand this error code: proxy: read response failed
from <server ip>:<port> (<server ip>)
it happens quite frequently when we do load test and on normal traffic it
happens but rarely. As load ramps up we see cpu utilization going up(not
hinting that this is reason) but we start seeing this error usually happens
once we cross about 300 user load.

we also have firewall in play between webserver and appserver.

After we see above error webserver jumps from one app server to another app
server. probably thinks original server is marked down. This has become very
comman with load going up hence I took thread dump to see what's happening and
I did not find any blocked threads. I can post whole thread dump if required
here is unique thread stack trace where most of threads are under load.

1) "ajp-bio-8009-exec-3396" daemon prio=10 tid=0x00007fa165b99800 nid=0x538b
runnable [0x00007fa069ad9000] 
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:312)
    at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:367)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:118)
    at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
    - locked <0x00000007d2998118> (a org.apache.tomcat.util.net.SocketWrapper)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)


2) "pool-8-thread-1281" prio=10 tid=0x00007fa0de140800 nid=0x5368 waiting on
condition [0x00007fa0721d9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000050e5f11f8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

3) "PooledThread[50]" prio=10 tid=0x00007fa0c4125000 nid=0x734 in Object.wait()
[0x00007fa0f064e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000006915ccf60> (a
de.hybris.platform.util.threadpool.PoolableThread)
    at java.lang.Object.wait(Object.java:503)
    at
de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246)
    - locked <0x00000006915ccf60> (a
de.hybris.platform.util.threadpool.PoolableThread)
    at
de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219)

4) "PooledThread[49]" prio=10 tid=0x00007fa0b801f800 nid=0x733 in Object.wait()
[0x00007fa066cab000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000653594388> (a
de.hybris.platform.util.threadpool.PoolableThread)
    at java.lang.Object.wait(Object.java:503)
    at
de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246)
    - locked <0x0000000653594388> (a
de.hybris.platform.util.threadpool.PoolableThread)
    at
de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219)

 5) "FD_SOCK client connection handler,hybris-broadcast,hybrisnode-2" daemon
prio=10 tid=0x00007fa0bc09d800 nid=0x67b7 runnable [0x00007fa06b9f8000]
    java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at
org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1120)
    at java.lang.Thread.run(Thread.java:745)
Here's our AJP connector config on tomcat

<Connector  protocol="AJP/1.3" 
  port="8009" 
  redirectPort="443" 
  useIPVHosts="true" 
  URIEncoding="UTF-8" 
  maxThreads="850" 
  minSpareThreads="100" 
  connectionTimeout="30000" 
  backlog="1000" 
  keepAliveTimeout="60000" 
  maxConnections="700" 
  acceptorThreadCount="4"/>
And below is apache conf

prefork.c
StartServers 20
MaxSpareServers 40
MinSpareServers 40
ServerLimit 700
MaxClients 700
IfModule
and lb.conf

BalancerMember ajp://<host_ip>:8009 route=app5 retry=60
ProxySet stickysession=JSESSIONID
ProxySet lbmethod=byrequests failonstatus=503,504

-- 
You are receiving this mail because:
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org


Mime
View raw message