hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CrystalCracker <purplehear...@hotmail.com>
Subject Re: BindException while using HttpClient under load
Date Tue, 25 May 2010 16:06:19 GMT

If I do netstat, I see thousands of TCP connections in TIME_WAIT state, which
means they are have been closed and are clearing up.

So to limit the no of connection, I use a single instance of HttpClient in
the following way. I think I would have to watch out for connections that
might have been stale, but overall does this sound about right? 

public class MyHttpClientFactory {
        private static MyHttpClientFactory instance = new
HttpClientFactory();
	private MultiThreadedHttpConnectionManager connectionManager;
	private HttpClient client;

	private HttpClientFactory() {
		init();
	}

	public static HttpClientFactory getInstance() {
		return instance;
	}

	public void init() {
		connectionManager = new MultiThreadedHttpConnectionManager();
		HttpConnectionManagerParams managerParams = new
HttpConnectionManagerParams();
		managerParams.setMaxTotalConnections(1000);
		connectionManager.setParams(managerParams);
		client = new HttpClient(connectionManager);
	}

	public HttpClient getHttpClient() {
		if (client != null) {
			return client;
                } else {
		    init();
		    return client;
                }
	}
}

MyHttpClientFactory.getInstance().getHttpClient();




Ken Krugler wrote:
> 
>  From looking at this code, it seems that you're creating a new  
> connection manager for each time the thread is run.
> 
> That seems kind of crazy. Normally you'd create a single HttpClient  
> instance, using the thread safe connection manager with support for  
> 1000 threads.
> 
> -- Ken
> 
> On May 24, 2010, at 4:16pm, CrystalCracker wrote:
> 
>>
>> I have got 1000 dedicated Java threads where each thread polls a
>> corresponding url every one second.
>>
>> Simplified version of the code is like this
>>
>> MyThread extends() {
>>   Node node;
>>   MyThread(Node node) {
>>      this.node = node;
>>   }
>>   public void run() {
>>       Poller.poll()
>>   }
>> }
>>
>> public class Poller {
>>    public static Node poll(Node node) {
>>        GetMethod method =  null;
>> 	try {
>> 	    HttpClient client = new HttpClient(new
>> SimpleHttpConnectionManager(true));
>> 	    client.getParams().setParameter("http.socket.timeout", 120000);
>> 	
>> 	    method = new GetMethod(node.getUrl());
>> 	    method.getParams().setSoTimeout(8000);
>>
>> 	    client.executeMethod(method);
>>
>> 	    method.getResponseHeaders();
>> 	    method.getResponseBody();
>> 	} catch (IOException ex) {
>>            ex.printStackTrace();
>>        } finally {
>>            method.releaseConnection();
>>        }
>>    }
>> }
>>
>> The threads are run one second:
>>
>> for (int i=0; i <1000; i++) {
>>    MyThread thread = threads.get(i) // threads  is a static field
>>    if(thread.isAlive()) {
>>        // If the previous thread is still running, let it run.
>>    } else {
>>        thread.start();
>>    }
>> }
>>
>> The problem is if I run the job every one second I get random  
>> exceptions
>> like these:
>>
>> java.net.BindException: Address already in use
>> INFO httpclient.HttpMethodDirector: I/O exception  
>> (java.net.BindException)
>> caught when processing request: Address already in use
>> INFO httpclient.HttpMethodDirector: Retrying request
>>
>> But if I run the job every 2 seconds or more, everything runs fine.
>>
>> I even tried shutting down the instance of  
>> SimpleHttpConnectionManager()
>> using shutDown() with no effect.
>>
>> It looks like if when myThread has finished running, the HttpClient it
>> started hasn't shutdown even though I told it to release conenction  
>> and/or
>> shutdown the manager.
>>
>> Any ideas?
>>
>>
>> -- 
>> View this message in context:
>> http://old.nabble.com/BindException-while-using-HttpClient-under-load-tp28662810p28662810.html
>> Sent from the HttpClient-User mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
> 
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c   w e b   m i n i n g
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/BindException-while-using-HttpClient-under-load-tp28662810p28670237.html
Sent from the HttpClient-User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message