Mailing-List: contact commons-httpclient-dev-help@jakarta.apache.org;
 run by ezmlm
Precedence: bulk
Reply-To: "Commons HttpClient Project"
 <commons-httpclient-dev@jakarta.apache.org>
Date: Fri, 10 Oct 2003 12:21:31 -0400
From: Eric Johnson <eric@tibco.com>
Subject: Re: Proposal: Configurable HTTP Response length limit
In-reply-to: <200310101752.43303.ck@rrzn.uni-hannover.de>
To: Commons HttpClient Project <commons-httpclient-dev@jakarta.apache.org>
Message-id: <3F86DC8B.5060504@tibco.com>
MIME-version: 1.0
Content-type: text/plain; charset=iso-8859-1; format=flowed
Content-transfer-encoding: 7BIT
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624
References: <200310101247.40166.ck@rrzn.uni-hannover.de>
 <200310101612.05457.ck@rrzn.uni-hannover.de> <3F86C9AD.4000404@tibco.com>
 <200310101752.43303.ck@rrzn.uni-hannover.de>

At a lower level, the potential failure points are undoubtedly there.  
Unless you could point to a real-world server that causes them, or a 
security bug that stems from them, any such fixes would be academic in 
nature.

Can you cause an out of memory failure while using 
getResponseBodyAsStream() to process a response from a *conformant* HTTP 
server?

Sure, I could construct a "chunked" response that indicated as a chunk 
size a number on the order of (16^(2^30)), which would consume all 
available memory, but then the question would be how to detect that and 
fail the connection sooner - due to a non-conformant server.  It would 
not be a question of merely prematurely terminating the chunk.

I don't see how ChunkedInputStream.exhaustInputStream() can actually 
consume all available memory.  Rather I see that if the server never 
stops sending data, it will simply never exit.  The latter condition 
suggests an abuse of the HTTP protocol, and an alternative "streaming" 
protocol would be more appropriate.

-Eric.

ck@rrzn.uni-hannover.de wrote:

> Eric,
>
>adding a maxBytes parameter to getResponseBody seems to be a resonable idea to 
>limit the actual number of bytes returned in the response body.
>
>Nevertheless, you would still have to fix some other flaws lying around at 
>lower levels that attempt to read until EOF or newline,  such as 
>ChunkedInputStream.close() and exhaustInputStream() - or 
>HttpParser.readRawline().
>
>I can easily provide test cases which will cause HttpClient to eat up all 
>available memory throwing an OutOfMemoryError because of reading and reading 
>from a never ending HTTP Response.
>
>I would regard this behaviour as a bug.
>
>The patch is intended as an "emergency brake" if methods at higher levels try 
>to read until EOF. It is up to the user whether to set this hard limit or 
>not.
>
>
>Christian
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org
>
>
>
>  
>


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org