hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasad Sethumadhavan" <PSethumadha...@sandcherry.com>
Subject RE: HttpClient under load
Date Thu, 03 Feb 2005 00:05:34 GMT
Hi Mike,
Thanks for your input. I am now using a single HttpClient instance and a
multi-threaded connection manager that is shared by all threads. It is
not failing any more.

I am trying to implement a "fetch timeout" feature using the http
client. Basically it means that if I have not received a complete
response within "n" seconds then I need to return a fetch timeout. 
Is there any built in mechanism in HttpClient for this or do I need to
have a monitor thread that calls HttpMethod.abort() after the timeout
period (if the method has not finished execution). What would be the
result of calling the abort method on the worker thread? (what
exceptions etc..). Please advise if there are any better ways to
implement the same.

Thanks in advance
Prasad

-----Original Message-----
From: Michael Becke [mailto:becke@u.washington.edu] 
Sent: Wednesday, February 02, 2005 6:39 AM
To: HttpClient User Discussion
Subject: Re: HttpClient under load

Hi Prasad,

Given the scenario you describe below you are more likely leaking one
connection/socket per request.  You definitely want to reuse instances
of HttpClient  and especially instances of a ConnectionManager.  You can
either use the multi-threaded connection manager from all threads or
have a dedicated SimpleHttpConnectionManager per thread.  Please have a
look at the performance guide for some more ideas
<http://jakarta.apache.org/commons/httpclient/3.0/performance.html>.

Also please note that calling manager.closeIdleConnections(1000L) only
closes connections that have already been idle for 1000ms.  It does not
continuously poll for idle connections, closing them when they've been
idle for 1000ms.

Mike

On Feb 1, 2005, at 11:14 PM, Prasad Sethumadhavan wrote:

> Hi,
> I need some help or pointers regarding using the HttpClient under 
> heavy load. I am currently using the HttpClient under moderate load 
> (50 concurrent threads). Each thread instatiates a new HttpClient 
> client, uses it and does the method.releaseConnection() before
exiting.
>
>         SimpleHttpConnectionManager manager = new 
> SimpleHttpConnectionManager();
>         manager.getParams().setStaleCheckingEnabled(true);
>         manager.closeIdleConnections(1000L);
>         manager.getParams().setConnectionTimeout(
>                 (int) DEFAULT_CONNECTION_TIMEOUT);
>
>         httpclient = new HttpClient(manager);
>         httpclient.getParams().setCookiePolicy(
>                 CookiePolicy.BROWSER_COMPATIBILITY);
>
> However when I run this setup under load, I keep getting socket 
> exceptions on the client side. I am using tomcat web server and I have

> increased the max-processor & access-count. I don't see any errors or 
> exception in the tomcat log.
>
> However I keep getting a lot of these client socket bind exceptions 
> after sometime. I guess it is happening due to the unavailability of 
> ports. I would like to know some best practices in using the 
> HttpClient under load. I am sure tha it is my bad, but wanted to get 
> some input from the user/dev community. Should I have a pool of 
> HttpClient objects and reuse them? Should I use the multi-threaded 
> connection manager? I need to run this under heavier load and need to 
> fix this soon. Any help, code snippets, or guidance would be greatly 
> appreciated. Please let me know if you need any other information.
>
> Thanks in advance
> Prasad
>
>
> Client StackTrace:
>
> [ERROR] 01 Feb 2005 20:43:56,177 NDC[] T[Thread-20] HTTP          HTTP
> I/O transport exception occurred
> java.net.BindException: Address already in use: connect
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> 	at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)
> 	at java.net.Socket.connect(Socket.java:507)
> 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> 	at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> o
> r
> Impl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:585)
> 	at
> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createS
> o
> c
> ket(ReflectionSocketFactory.java:128)
> 	at
> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cr
> e
> a
> teSocket(DefaultProtocolSocketFactory.java:124)
> 	at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:

> 70
> 5)
> 	at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
> M
> e
> thodDirector.java:359)
> 	at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMet
> h
> o
> dDirector.java:168)
> 	at
>
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java: 
> 3
> 93)
> 	at
>
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java: 
> 3
> 24)
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:  
> httpclient-user-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


Mime
View raw message