Return-Path: 
 <httpclient-users-return-11981-apmail-hc-httpclient-users-archive=hc.apache.org@hc.apache.org>
X-Original-To: apmail-hc-httpclient-users-archive@www.apache.org
Delivered-To: apmail-hc-httpclient-users-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DCE68106B3
	for <apmail-hc-httpclient-users-archive@www.apache.org>;
 Tue, 11 Feb 2014 04:08:09 +0000 (UTC)
Received: (qmail 44172 invoked by uid 500); 11 Feb 2014 04:08:09 -0000
Delivered-To: apmail-hc-httpclient-users-archive@hc.apache.org
Received: (qmail 43618 invoked by uid 500); 11 Feb 2014 04:08:01 -0000
Mailing-List: contact httpclient-users-help@hc.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:httpclient-users-help@hc.apache.org>
List-Unsubscribe: <mailto:httpclient-users-unsubscribe@hc.apache.org>
List-Post: <mailto:httpclient-users@hc.apache.org>
List-Id: <httpclient-users.hc.apache.org>
Reply-To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Delivered-To: mailing list httpclient-users@hc.apache.org
Received: (qmail 43609 invoked by uid 99); 11 Feb 2014 04:07:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Feb 2014 04:07:59 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of fancyerii@gmail.com
 designates 209.85.217.178 as permitted sender)
Received: from [209.85.217.178] (HELO mail-lb0-f178.google.com)
 (209.85.217.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Feb 2014 04:07:53 +0000
Received: by mail-lb0-f178.google.com with SMTP id u14so5377729lbd.23
        for <httpclient-users@hc.apache.org>;
 Mon, 10 Feb 2014 20:07:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=bf3blqZCWtjafea+dpuwqv4YVygWS23+u+g6DIhDUOM=;
        b=vwP87nuovhbptB7//7ROcVAKcm1U5kbwfz2wUYCi7WTozrEe89GmDNe5hbOMEjWoGI
         NjqepCdEwe7BU/OKxbsaZnDyrUrwlf5hCNPWnPuBNYJQ/EGmN+y1fK8Udm/VqO8FRJDi
         ahAr1isbHg40QXDq00cBcSVRCs7l2MDOR7wD7HHRTUQHInG6tGLQmXl/F65JFJZQQWAc
         eZJCqtjo0oSFE7lUSTIcMwgyBZnUO12vDjRc0kkRbssHrt08HG3ODX1P8cGWE4ECKYS0
         VxGwj1sNXS04FTrn3w7pJefvdN5+liW9mTCKNYuuRJnSbgKg3gEkE5WQse+wZDPzMsdm
         /fag==
MIME-Version: 1.0
X-Received: by 10.152.5.199 with SMTP id u7mr25050609lau.16.1392091651989;
 Mon, 10 Feb 2014 20:07:31 -0800 (PST)
Received: by 10.112.2.166 with HTTP; Mon, 10 Feb 2014 20:07:31 -0800 (PST)
Date: Tue, 11 Feb 2014 12:07:31 +0800
Message-ID: 
 <CAFAd71VqAzzMu3DZ+_qcAs2a7DfZShqzrN0-E26zo=FWsHmbSQ@mail.gmail.com>
Subject: OOM problem
From: Li Li <fancyerii@gmail.com>
To: httpclient-users@hc.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I am using httpclient 4.3 to crawl webpages.
I start 200 threads and PoolingHttpClientConnectionManager with
totalMax 1000 and perHostMax 5
I give java 2GB memory and one thread throws an exception(others still
running, this thread is dead)

Exception in thread "Thread-156" java.lang.OutOfMemoryError: Java heap space
        at org.apache.http.util.ByteArrayBuffer.<init>(ByteArrayBuffer.java:56)
        at org.apache.http.util.EntityUtils.toByteArray(EntityUtils.java:133)
        at com.founder.httpclientfetcher.HttpClientFetcher$3.handleResponse(HttpClientFetcher.java:221)
        at com.founder.httpclientfetcher.HttpClientFetcher$3.handleResponse(HttpClientFetcher.java:211)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:218)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:136)
        at com.founder.httpclientfetcher.HttpClientFetcher.httpGet(HttpClientFetcher.java:233)
        at com.founder.vcfetcher.CrawlWorker.getContent(CrawlWorker.java:198)
        at com.founder.vcfetcher.CrawlWorker.doWork(CrawlWorker.java:134)
        at com.founder.vcfetcher.CrawlWorker.run(CrawlWorker.java:231)

does it mean my code has some memory leak probelm?

my codes:
public String httpGet(String url) throws Exception {
if (!isValid)
throw new RuntimeException("not valid now, you should init first");
HttpGet httpget = new HttpGet(url);

// Create a custom response handler
ResponseHandler<String> responseHandler = new ResponseHandler<String>() {

public String handleResponse(final HttpResponse response)
throws ClientProtocolException, IOException {
int status = response.getStatusLine().getStatusCode();
if (status >= 200 && status < 300) {
HttpEntity entity = response.getEntity();
if (entity == null)
return null;

byte[] bytes = EntityUtils.toByteArray(entity);
String charSet = CharsetDetector.getCharset(bytes);

return new String(bytes, charSet);
} else {
throw new ClientProtocolException(
"Unexpected response status: " + status);
}
}

};

String responseBody = client.execute(httpget, responseHandler);
return responseBody;
}

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org