hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suladna <sula...@yahoo.com>
Subject Re: My HttpClient gives me a memory leak
Date Thu, 04 Sep 2008 17:23:06 GMT
Hm I don't think I keep any content but it is worth a try. Thanks.


--- On Thu, 9/4/08, Tomek Maciejewski <tomek.maciejewski@gmail.com> wrote:

From: Tomek Maciejewski <tomek.maciejewski@gmail.com>
Subject: Re: My HttpClient gives me a memory leak
To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Date: Thursday, September 4, 2008, 1:27 PM

Hi Suladna,

What do you do with the content of the site during the 'processing 
html'. It is possible that if you download few hundreds of sites and 
keep content (or some kind of parsed object representation) of every 
site in memory, then you can have too low heap size declared. You can 
declare the maximum heap size of your JVM by adding -Xmx argument , i.e. 
by seting -Xmx512M you defines heap size as 512MB. But I'm not sure if 
it is a reason of your problem.

Tomek


Suladna wrote:
> Hi
>  
> I have used the following tutorial to bring html-code from websites to 
> my java program:
>  
>
http://svn.apache.org/repos/asf/httpcomponents/httpclient/trunk/module-client/src/examples/org/apache/http/examples/client/ClientExecuteDirect.java
>  
> The websites which I connect to all start with www.xxxxxxxxx.se 
> <http://www.xxxxxxxxx.se/> *  but they have different endings.
>  
> So I start by making one HttpHost and defining a HttpEntity and 
> HttpRespons
>  
> final HttpHost target = new HttpHost("www.xxxxxxx.se", 80,
"http");
> HttpEntity entity = null;
> HttpResponse rsp = null;
>
>  
> After that I use a loop to connect to each subsite. This is what I 
> write in the loop:
>  
> HttpRequest req = createRequest(urlEnding);
> rsp = client.execute(target, req);
>
> entity = rsp.getEntity();
>
> String[] line = EntityUtils.toString(entity).split("\n");
//this info 
> is used by the program
>
> *if* (entity != *null*){
>
> entity.consumeContent();
>
> }
>  
> It basically works fine, but the problem is that I get a memory leak.. 
> after having brought the code from a few hundred websites I get a 
> java.lang.OutOfMemoryError: Java heap space. Is there anything I can 
> do in the loop except entity.consumeConent() to prevent this from 
> happening?
>  
> I attach my java code as a text file. It is just a slight modification 
> of the code in the tutorial.
>  
> * this is not the real domain.. if anyone wants to know the real 
> domain, please send me a private e-mail
>
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org




      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message