hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasad Sethumadhavan" <PSethumadha...@sandcherry.com>
Subject RE: HttpClient under load
Date Thu, 03 Feb 2005 00:32:00 GMT
Thanks Julius. I need to handle slow servers too.  
Actually I tried using the methodParams.setSoTimeout(5000)//5 seconds
with 3.0beta1 and the servlet sleeps for 30 seconds before writing some
response. However no exception was thrown. Any ideas??

-----Original Message-----
From: Julius Davies [mailto:juliusdavies@cucbc.com] 
Sent: Wednesday, February 02, 2005 5:22 PM
To: HttpClient User Discussion
Subject: RE: HttpClient under load

Hi, Prasad,

You can use this to catch sockets that stop writing back to your stream:

==========================
HttpClient client = new HttpClient();
client.setTimeout( 60000 ); // socket timeout of 60 seconds
==========================

I don't think it will catch sockets that write back slowly (e.g. 1
character per second).  For that you probably need to implement your own
timeout.


yours,

Julius

On Wed, 2005-02-02 at 17:05 -0700, Prasad Sethumadhavan wrote:
> Hi Mike,
> Thanks for your input. I am now using a single HttpClient instance and

> a multi-threaded connection manager that is shared by all threads. It 
> is not failing any more.
> 
> I am trying to implement a "fetch timeout" feature using the http 
> client. Basically it means that if I have not received a complete 
> response within "n" seconds then I need to return a fetch timeout.
> Is there any built in mechanism in HttpClient for this or do I need to

> have a monitor thread that calls HttpMethod.abort() after the timeout 
> period (if the method has not finished execution). What would be the 
> result of calling the abort method on the worker thread? (what 
> exceptions etc..). Please advise if there are any better ways to 
> implement the same.
> 
> Thanks in advance
> Prasad
> 
> -----Original Message-----
> From: Michael Becke [mailto:becke@u.washington.edu]
> Sent: Wednesday, February 02, 2005 6:39 AM
> To: HttpClient User Discussion
> Subject: Re: HttpClient under load
> 
> Hi Prasad,
> 
> Given the scenario you describe below you are more likely leaking one 
> connection/socket per request.  You definitely want to reuse instances

> of HttpClient  and especially instances of a ConnectionManager.  You 
> can either use the multi-threaded connection manager from all threads 
> or have a dedicated SimpleHttpConnectionManager per thread.  Please 
> have a look at the performance guide for some more ideas 
> <http://jakarta.apache.org/commons/httpclient/3.0/performance.html>.
> 
> Also please note that calling manager.closeIdleConnections(1000L) only

> closes connections that have already been idle for 1000ms.  It does 
> not continuously poll for idle connections, closing them when they've 
> been idle for 1000ms.
> 
> Mike
> 
> On Feb 1, 2005, at 11:14 PM, Prasad Sethumadhavan wrote:
> 
> > Hi,
> > I need some help or pointers regarding using the HttpClient under 
> > heavy load. I am currently using the HttpClient under moderate load 
> > (50 concurrent threads). Each thread instatiates a new HttpClient 
> > client, uses it and does the method.releaseConnection() before
> exiting.
> >
> >         SimpleHttpConnectionManager manager = new 
> > SimpleHttpConnectionManager();
> >         manager.getParams().setStaleCheckingEnabled(true);
> >         manager.closeIdleConnections(1000L);
> >         manager.getParams().setConnectionTimeout(
> >                 (int) DEFAULT_CONNECTION_TIMEOUT);
> >
> >         httpclient = new HttpClient(manager);
> >         httpclient.getParams().setCookiePolicy(
> >                 CookiePolicy.BROWSER_COMPATIBILITY);
> >
> > However when I run this setup under load, I keep getting socket 
> > exceptions on the client side. I am using tomcat web server and I 
> > have
> 
> > increased the max-processor & access-count. I don't see any errors 
> > or exception in the tomcat log.
> >
> > However I keep getting a lot of these client socket bind exceptions 
> > after sometime. I guess it is happening due to the unavailability of

> > ports. I would like to know some best practices in using the 
> > HttpClient under load. I am sure tha it is my bad, but wanted to get

> > some input from the user/dev community. Should I have a pool of 
> > HttpClient objects and reuse them? Should I use the multi-threaded 
> > connection manager? I need to run this under heavier load and need 
> > to fix this soon. Any help, code snippets, or guidance would be 
> > greatly appreciated. Please let me know if you need any other
information.
> >
> > Thanks in advance
> > Prasad
> >
> >
> > Client StackTrace:
> >
> > [ERROR] 01 Feb 2005 20:43:56,177 NDC[] T[Thread-20] HTTP
HTTP
> > I/O transport exception occurred
> > java.net.BindException: Address already in use: connect
> > 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> > 	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> > 	at
> > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> > 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> > 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)
> > 	at java.net.Socket.connect(Socket.java:507)
> > 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > 	at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> > ss
> > o
> > r
> > Impl.java:25)
> > 	at java.lang.reflect.Method.invoke(Method.java:585)
> > 	at
> > org.apache.commons.httpclient.protocol.ReflectionSocketFactory.creat
> > eS
> > o
> > c
> > ket(ReflectionSocketFactory.java:128)
> > 	at
> > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.
> > cr
> > e
> > a
> > teSocket(DefaultProtocolSocketFactory.java:124)
> > 	at
> >
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:
> 
> > 70
> > 5)
> > 	at
> > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Ht
> > tp
> > M
> > e
> > thodDirector.java:359)
> > 	at
> > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpM
> > et
> > h
> > o
> > dDirector.java:168)
> > 	at
> >
>
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java: 
> > 3
> > 93)
> > 	at
> >
>
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java: 
> > 3
> > 24)
> >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: 
> > httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:  
> > httpclient-user-help@jakarta.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> httpclient-user-help@jakarta.apache.org
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: 
> httpclient-user-help@jakarta.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


Mime
View raw message