hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Li <fancye...@gmail.com>
Subject Re: OOM problem
Date Tue, 11 Feb 2014 04:11:42 GMT
jmap result:
Debugger attached successfully.
Server compiler detected.
JVM version is 22.1-b02

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 2147483648 (2048.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 85983232 (82.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 599130112 (571.375MB)
   used     = 143482424 (136.83550262451172MB)
   free     = 455647688 (434.5394973754883MB)
   23.94845812723898% used
>From Space:
   capacity = 41811968 (39.875MB)
   used     = 41757744 (39.82328796386719MB)
   free     = 54224 (0.0517120361328125MB)
   99.87031464292711% used
To Space:
   capacity = 57671680 (55.0MB)
   used     = 0 (0.0MB)
   free     = 57671680 (55.0MB)
   0.0% used
PS Old Generation
   capacity = 1009254400 (962.5MB)
   used     = 557209200 (531.3961029052734MB)
   free     = 452045200 (431.10389709472656MB)
   55.209984717431006% used
PS Perm Generation
   capacity = 34275328 (32.6875MB)
   used     = 24751016 (23.604408264160156MB)
   free     = 9524312 (9.083091735839844MB)
   72.21233885785134% used

On Tue, Feb 11, 2014 at 12:07 PM, Li Li <fancyerii@gmail.com> wrote:
> I am using httpclient 4.3 to crawl webpages.
> I start 200 threads and PoolingHttpClientConnectionManager with
> totalMax 1000 and perHostMax 5
> I give java 2GB memory and one thread throws an exception(others still
> running, this thread is dead)
>
> Exception in thread "Thread-156" java.lang.OutOfMemoryError: Java heap space
>         at org.apache.http.util.ByteArrayBuffer.<init>(ByteArrayBuffer.java:56)
>         at org.apache.http.util.EntityUtils.toByteArray(EntityUtils.java:133)
>         at com.founder.httpclientfetcher.HttpClientFetcher$3.handleResponse(HttpClientFetcher.java:221)
>         at com.founder.httpclientfetcher.HttpClientFetcher$3.handleResponse(HttpClientFetcher.java:211)
>         at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:218)
>         at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160)
>         at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:136)
>         at com.founder.httpclientfetcher.HttpClientFetcher.httpGet(HttpClientFetcher.java:233)
>         at com.founder.vcfetcher.CrawlWorker.getContent(CrawlWorker.java:198)
>         at com.founder.vcfetcher.CrawlWorker.doWork(CrawlWorker.java:134)
>         at com.founder.vcfetcher.CrawlWorker.run(CrawlWorker.java:231)
>
> does it mean my code has some memory leak probelm?
>
> my codes:
> public String httpGet(String url) throws Exception {
> if (!isValid)
> throw new RuntimeException("not valid now, you should init first");
> HttpGet httpget = new HttpGet(url);
>
> // Create a custom response handler
> ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
>
> public String handleResponse(final HttpResponse response)
> throws ClientProtocolException, IOException {
> int status = response.getStatusLine().getStatusCode();
> if (status >= 200 && status < 300) {
> HttpEntity entity = response.getEntity();
> if (entity == null)
> return null;
>
> byte[] bytes = EntityUtils.toByteArray(entity);
> String charSet = CharsetDetector.getCharset(bytes);
>
> return new String(bytes, charSet);
> } else {
> throw new ClientProtocolException(
> "Unexpected response status: " + status);
> }
> }
>
> };
>
> String responseBody = client.execute(httpget, responseHandler);
> return responseBody;
> }

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message