Mailing-List: contact dev-help@httpd.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@httpd.apache.org
Received-SPF: pass (nike.apache.org: domain of bongjae.chang@gmail.com
 designates 209.85.161.173 as permitted sender)
User-Agent: Microsoft-MacOutlook/14.2.2.120421
Date: Fri, 25 May 2012 12:04:25 +0900
Subject: Re: CLOSE_WAIT problem
From: Bongjae Chang <bongjae.chang@gmail.com>
To: <dev@httpd.apache.org>
Message-ID: <CBE517F4.1E6C7%bongjae.chang@gmail.com>
Thread-Topic: CLOSE_WAIT problem
In-Reply-To: 
 <CAGnVnBnr4TM7=p5YWjQ-ZF_xyERek3FAtvY2-x-jtZjoxb2yVg@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

Hi Sridhar,

Thank you for your kind words.

First, I am sorry that I also sent this similar mail to users mailing
again before I received your mail.

I received an answer from Elic that the closed conn will be closed when it
will be reused as your words.

Though I sent the following question to users mailing, I would like to
discuss more if this content is suitable for devs mailing.

IMHO, as soon as connections are closed from backend, if Apache can detect
closed connections of backend and close them explicitly, CLOSE_WAITS can
disappear as soon as possible before Apache tries to reuse them.

And I read the default max number which will be allowed to backend is
ThreadsPerChild directive and users can set the max or ThreadsPerChild to
be a big number. Then, I think many CLOSE_WAITs can be piled up. It's a
waste of Fds.

 
Actually, I was afraid of the following situation.
 

1. Apache has many backend connections(i.g. 8080 backend port)
2. Backend server should be replaced for maintenace
3. So Apache connects new backend server(i.g. 9090 backend port)
dynamically without reboot and old backend server will be shut down
4. Then all old backend connections(i.g. 8080 backend port) will become to
be CLOSE_WAIT state for long time.

When I tested this, Apache didn't grab old connections(8080 backend) from
a resource pool because Apache was only interested in new pool(9090 port).
So CLOSE_WAITs(8080 port) remained forever.

When I tested mod_jk, as your words, mod_jk had a background thread and
pinged, so CLOSE_WAITs could be cleaned up periodically.

But, I have a question. Can't Apache detect the closed connection as soon
as the backend connection is closed?
I am sorry that I don't review codes yet so I don't know if Apache uses
blocking I/O or NIO for backend.

In Grizzly(http://grizzly.java.net/) which is NIO framework based on
Java(and I participated in), I can see the selector is waked with op_read
when the peer connection is closed, then read() will return -1 and closes
it.

Is the watcher thread which is going through all of the connections
looking to see if they have been closed by the peer only solution?

Thanks!


Regards,
Bongjae Chang


On 5/24/12 10:42 PM, "sridhar basam" <sri@basam.org> wrote:

>On Wed, May 23, 2012 at 1:35 AM, Bongjae Chang <bongjae.chang@gmail.com>
>wrote:
>> Hi all,
>>
>> When I tested mod_proxy + mod_proxy_balancer + (mod_proxy_ajp or
>> mod_proxy_http) with worker mpm, I always met CLOSE_WAIT state in apache
>> proxy side.
>>
>> I tested the following scenario.
>>
>> - Sending a request at the browser -> apache,  mod proxy -> ajp or http
>>java
>> server(1)
>> - Normally, browser received the response correctly and the connection
>>state
>> was ESTABLISHED.
>> - And java server closed the ajp or http connection with timeout(Or
>> terminate java server forcibly).
>> - Then apache proxy machine always had the CLOSE_WAIT state about the
>> connection.
>>
>> It seemed that the apache proxy modules did not try to close the invalid
>> socket which had been already closed at the peer side(the backend java
>> server side).
>>
>> Perhaps is it already the known issue?
>
>I believe this is normal. Apache keeps a pool of connections to the
>backend to re-use. There isn't an active thread going through all of
>the connections looking to see if they have been closed by the peer.
>The proxy module will check the state of a connection after grabbing
>it from a resource pool before it re-uses it so the connection in
>close_wait will eventually get closed and re-opened. Worse case,  the
>number of close_wait connections would be capped at an upper limit
>based on your configs and your upstream server.
>
> Sridhar
>
>
>
>
>>
>> Please give me some advice if I am misunderstanding something.
>>
>> Thanks!
>>
>> --
>> apache version: 2.4.2
>> backend java server: grizzly http/ajp, playframework(maybe it uses
>>netty)
>> os: Apache(Linux 2.6.18 x86_64), BackendServer(MacOs)
>> --
>>
>> Regards,
>> Bongjae Chang