Mailing-List: contact httpclient-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient Project" <httpclient-dev@jakarta.apache.org>
Received-SPF: pass (asf.osuosl.org: domain of odi@odi.ch designates
 62.65.128.20 as permitted sender)
Message-ID: <446AD0C5.1080700@odi.ch>
Date: Wed, 17 May 2006 09:29:09 +0200
From: =?UTF-8?B?T3J0d2luIEdsw7xjaw==?= <odi@odi.ch>
User-Agent: Thunderbird 1.5.0.2 (Windows/20060308)
MIME-Version: 1.0
To: HttpClient Project <httpclient-dev@jakarta.apache.org>
Subject: Re: [jira] Resolved: (HTTPCORE-3) HttpParser triggers unfriendly
 OutOfMemoryError on challenging input
References: <23689632.1147727886165.JavaMail.jira@brutus>
 <446A5108.4050706@archive.org>
In-Reply-To: <446A5108.4050706@archive.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit


Gordon Mohr wrote:
> If I understand the HttpCore code properly, there is no direct facility 
> for protecting against the OOME in the code -- just a chance to hook in 
> a theoretical alternate implementation that would address the problem.
> 
> Is that correct?

Yes.

> To use the HttpCore-4.0 facility, it appears I would create my own 
> HttpDataReceiver implementation which keeps a count of the bytes it 
> shovels & throws an IO or HTTP exception when some count is exceeded; 
> create a factory that makes such receivers; install that factory into 
> each HttpClientConnection instance before it begins receiving data.

Correct.

> This could work, but seems a roundabout and obscure approach. The really 
> valuable feature would be for OOME-resistance -- and friendly, usable 
> indicators that extreme content has been encountered -- to be features 
> of the library. It's require a switch or paramter to enable, rather than 
> patching in custom/third-party code.

An OOM Exception is exactly that: a friendly usable indicator that an 
extreme condition has been encountered. I am personally against another 
mechanism that tries to somehow monitor heap memory.

I agree that by sending large header information a malicious server 
could DOS a spider based on HttpCore. So it makes sense for this use 
case to include a protection parameter that sets an upper limit to the 
information in headers. A default of 100 KB should be enough for the 
real world. This is the easy solution.

Another possibility is to use a header stream like we do for the message 
body. This would mean a refactoring and probably cause painful client 
code. We would still have to protect ourselves against long header lines 
which is again not easy.

This all boils down to a missing feature: byte counting in the connection.

> Is there a summary of expected dates of Core-4.0/Client-4.0 release 
> somewhere, or any assessments of how the 4.0 codebases match up against 
> 3.0 features? (Is it reasonable for an HttpClient-3.0-using project to 
> consider transitioning to the 4.0 codebase(s)?)

This is the only plan there is:
http://wiki.apache.org/jakarta-httpclient/HttpComponentsProjectRoadmap


> - Gordon @ IA

Ortwin

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org