httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "azurIt" <azu...@pobox.sk>
Subject Re: Taking 100% CPU when network is throttled
Date Mon, 01 Aug 2011 13:10:55 GMT

______________________________________________________________
> Od: "Eric Covener" <covener@gmail.com>
> Komu: dev@httpd.apache.org
> Dátum: 01.08.2011 15:02
> Predmet: Re: Taking 100% CPU when network is throttled
>
>On Mon, Aug 1, 2011 at 8:45 AM, azurIt <azurit@pobox.sk> wrote:
>>
>> Hi,
>>
>> i came accross a serious problem with Apache server when it is NOT able to read/write
from/to network immediatelly.
>>
>> We are using network throttling patches in our linux kernel wihch are able to emulate
low network throughput (we are using this to throttle users). When a process reaches it's
limit, read/write will result in EAGAIN which is completely ok cos it's standard behavior.
Problem is that Apache is not able to correctly process this and is trying to read/write in
a loop WITHOUT any delay - this is, of course, resulting in 100% CPU consumption. This bug
can lead to DoS of the whole server.
>>
>> Here is the strace output when Apache's child is throttled:
>>
>> writev(6409, [{"n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n"..., 2900}], 1) = -1 EAGAIN (Resource
temporarily unavailable)
>> poll([{fd=6409, events=POLLOUT}], 1, 100000) = 1 ([{fd=6409, revents=POLLOUT}])
>> writev(6409, [{"n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n"..., 2900}], 1) = -1 EAGAIN (Resource
temporarily unavailable)
>> poll([{fd=6409, events=POLLOUT}], 1, 100000) = 1 ([{fd=6409, revents=POLLOUT}])
>> writev(6409, [{"n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n"..., 2900}], 1) = -1 EAGAIN (Resource
temporarily unavailable)
>> poll([{fd=6409, events=POLLOUT}], 1, 100000) = 1 ([{fd=6409, revents=POLLOUT}])
>> writev(6409, [{"n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n"..., 2900}], 1) = -1 EAGAIN (Resource
temporarily unavailable)
>> poll([{fd=6409, events=POLLOUT}], 1, 100000) = 1 ([{fd=6409, revents=POLLOUT}])
>> writev(6409, [{"n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n1n"..., 2900}], 1) = -1 EAGAIN (Resource
temporarily unavailable)
>> poll([{fd=6409, events=POLLOUT}], 1, 100000) = 1 ([{fd=6409, revents=POLLOUT}])
>>
>> ..and so on until it is able to send data again.
>>
>> I suggest to insert a little delay everytime an EAGAIN is returned.
>
>Looks like your kernel patches cause poll() to tell Apache the socket
>is immediately writable even when the kernel will return EAGAIN on a
>subsequent write.
>
>I'd expect without the patches, and with a client that doesn't read
>the [large] response, you won't see the behavior and poll() will just
>block (not spin) until you do some reading.
>


Thank you for your opinion, we will dig deeper into this.

Have a nice day.

azur

Mime
View raw message