Subject: Re: denial of service attack on web servers (fwd)
To: new-httpd@hyperreal.com
Date: Sat, 19 Oct 1996 16:34:53 +0100 (BST)
From: Ben Laurie <ben@gonzo.ben.algroup.co.uk>
In-Reply-To: <199610191535.QAA06572> from "Rob Hartill" at Oct 19,
 96 04:35:07 pm
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Message-ID: <9610191634.aa20614@gonzo.ben.algroup.co.uk>
Sender: owner-new-httpd@apache.org
Precedence: bulk
Reply-To: new-httpd@hyperreal.com

Rob Hartill wrote:
> 
> 
> I've perhaps missed this guy's point, but he seems to be saying that
> people can cause a denial of service by holding connections open, and that
> it lasts for 240s at the TCP level.
> 
> Apache has a default timeout of 400s, so does this make the slightest bit
> of difference ?
> 
> Note, this was sent to CERT, so a swift response is called for before
> the vicious rumours spread.

This denial of service attack works for all TCP based protocols where the
server is the one that does a listen. If they don't use a timeout, it works
extremely well. If they do, then, as the man says, you get the resource
consumed for timeout+FIN_WAIT2 time. This is not news, of course, which is why
Sun (and others) have a page saying what to do about it.

So, this has nothing really to do with webservers, per se, and everything to do
with the design of TCP. IMHO, setting the FIN_WAIT2 time very small is unlikely
to introduce problems for well-behaved clients.

Cheers,

Ben.

> 
> rob
> 
> ----- Forwarded message from John Winans -----
> 
> From: John Winans <winans@ganymede.net>
> Date: Sat, 19 Oct 1996 10:20:03 -0500
> Message-Id: <199610191520.KAA03480@zikzak.ganymede.net>
> To: cert@cert.org
> Subject: denial of service attack on web servers
> Cc: apache-bugs@apache.org, ses@unc.edu, timbl@w3.org
> 
> 
> 
> I believe that it would not be hard to take advantage of the situation I 
> describe below to create a denial of service attack that can cause problems 
> on a web server.  Depending on the server's implementation of TCP/IP these 
> problems could reach beyond the web server processes themselves.  The result
> at could range from a temporary denial of service to a server system crash.
> 
> --John Winans 
> President
> Ganymede Corporation
> 
> 
> =============================================================================
> 
> 
> This past summer I researched a hanging problem that is talked about in the
> apache 'known bugs' section.  I don't entirely agree with the posted 'lab 
> report' in that it is simply Netscape bug.
> 
> I ran tests from Netscape 2.x on Solaris, W95, and NetBSD making hits to
> Apache 1.1b3 running on plain NetBSD 1.1.  The stuff ruinning on W95 caused a 
> hanging problem.  What I observed was that after a keep-alive connection had 
> timed out, apache closed the socket on the server and it went into the 
> FIN_WAIT2 state and stayed there for a very long time.  FIN_WAIT2 is the 
> state of a TCP connection that is only 1/2 closed.  It can only entered by 
> the side of the connection that performs the active close on a TCP socket... 
> the web server. (See page 241 of "TCP/IP Illustrated Volume 1" by Richard 
> Stevens and run netstat on your web server to watch the socket state 
> transitions.)
> 
> The 'bug' in this specific case is (probably) that a select statement in 
> Netscape (prior to the supposedly fixed version 3.x) that monitors activity 
> on its sockets is not reporting the EOF condition or Netscape is simply not 
> closing the socket at that time.  This results in a situation where Netscape
> (apparantly) thinks it can still use the 1/2 open socket.
> 
> What bothered me was that this apparant "browser bug" was costing me resources 
> on my web server.  While this does appear to be a bug in Netscape and an annoyance to a user accessing my web server, it is also representative of a greater 
> problem in the way we use the TCP protocol in HTTP.  In fact, I'll even go one 
> further... it is a problem in general with high transaction-rate servers where 
> the server performs the active-close operation on TCP connections (HTTP 
> keep-alive or not.)
> 
> 
> 
> I invested a few hours in this problem and found that this is a taste of a 
> far greater problem.  This problem could be exploited to implement a denial 
> of service attack on a web server.
> 
> Chapter 1 and pages 29 and 30 in "TCP Illustrated Volume 3", by Richard Stevens 
> include a technical discussion as to why this can be a serious problem.
> 
> There is also a general discussion of this situation at:
> 
> 	http://sunsite.unc.edu/mdma-release/http-prob.html
> 
> At the bottom you will find the following summary:
> 
>   One scalability problem caused by the single request per connection
>   paradigm occurs due to TCP's TIME_WAIT state. When a server closes a
>   TCP, connection, it is required to keep information about that  
>   connection for a period of time, in case a delayed packet turns up and 
>   sabotages a new incarnation of the connection. 
> 
>   The recommended time to keep this information is 240 seconds. Because 
>   of this, a server will leave some resources allocated for every single  
>   connection closed in the past four minutes. On a busy server, this can
>   add up to thousands of control blocks. 
> 
> 
> This summary is based on a web server and clients that are operating PROPERLY.  
> If someone wanted to, they could write a web walker that exploited the use of
> the FIN_WAITx states by opening sockets to web servers and simply never closing 
> them (or more accurately, never sending a FIN.)  When sockets get stuck in 
> FIN_WAITx2 they take far longer to time out than they do when they are in
> TIME_WAIT.  A flood of connections that are left stuck in FIN_WAITx could 
> consume a great deal of kernel memory and result in unexpected operation of the 
> networking subsystem or the kernel itself.
> 
> 
> I suspect that it may be possible to work with w3.org to push the internet 
> community into a new generation of safer HTTP protocol(s.)  After all, it is 
> groups like Apache and w3.org that delivered the free and useful software that 
> drove much of the web in the first place.  If an improvement to the security 
> and reliability of system operation can be made and demonstrated, the 'net'
> will have to buy into it or risk attacks (and bugs) that cause the types
> of service problems I describe.  
> 
> 
> -- 
> ! John Winans                     Ganymede Corp.                        !
> ! winans@ganymede.net             http://www.ganymede.net               !
> !                                                                       !
> !"The large print giveth, and the small print taketh away." - Tom Waits !
> 
> ----- End of forwarded message from John Winans -----
> 

-- 
Ben Laurie                  Phone: +44 (181) 994 6435
Freelance Consultant and    Fax:   +44 (181) 994 6472
Technical Director          Email: ben@algroup.co.uk
A.L. Digital Ltd,           URL: http://www.algroup.co.uk
London, England.            Apache Group member (http://www.apache.org)