hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julius Davies <juliusdav...@cucbc.com>
Subject RE: HttpClient under load
Date Thu, 03 Feb 2005 00:22:10 GMT
Hi, Prasad,

You can use this to catch sockets that stop writing back to your stream:

==========================
HttpClient client = new HttpClient();
client.setTimeout( 60000 ); // socket timeout of 60 seconds
==========================

I don't think it will catch sockets that write back slowly (e.g. 1
character per second).  For that you probably need to implement your own
timeout.


yours,

Julius

On Wed, 2005-02-02 at 17:05 -0700, Prasad Sethumadhavan wrote:
> Hi Mike,
> Thanks for your input. I am now using a single HttpClient instance and a
> multi-threaded connection manager that is shared by all threads. It is
> not failing any more.
> 
> I am trying to implement a "fetch timeout" feature using the http
> client. Basically it means that if I have not received a complete
> response within "n" seconds then I need to return a fetch timeout. 
> Is there any built in mechanism in HttpClient for this or do I need to
> have a monitor thread that calls HttpMethod.abort() after the timeout
> period (if the method has not finished execution). What would be the
> result of calling the abort method on the worker thread? (what
> exceptions etc..). Please advise if there are any better ways to
> implement the same.
> 
> Thanks in advance
> Prasad
> 
> -----Original Message-----
> From: Michael Becke [mailto:becke@u.washington.edu] 
> Sent: Wednesday, February 02, 2005 6:39 AM
> To: HttpClient User Discussion
> Subject: Re: HttpClient under load
> 
> Hi Prasad,
> 
> Given the scenario you describe below you are more likely leaking one
> connection/socket per request.  You definitely want to reuse instances
> of HttpClient  and especially instances of a ConnectionManager.  You can
> either use the multi-threaded connection manager from all threads or
> have a dedicated SimpleHttpConnectionManager per thread.  Please have a
> look at the performance guide for some more ideas
> <http://jakarta.apache.org/commons/httpclient/3.0/performance.html>.
> 
> Also please note that calling manager.closeIdleConnections(1000L) only
> closes connections that have already been idle for 1000ms.  It does not
> continuously poll for idle connections, closing them when they've been
> idle for 1000ms.
> 
> Mike
> 
> On Feb 1, 2005, at 11:14 PM, Prasad Sethumadhavan wrote:
> 
> > Hi,
> > I need some help or pointers regarding using the HttpClient under 
> > heavy load. I am currently using the HttpClient under moderate load 
> > (50 concurrent threads). Each thread instatiates a new HttpClient 
> > client, uses it and does the method.releaseConnection() before
> exiting.
> >
> >         SimpleHttpConnectionManager manager = new 
> > SimpleHttpConnectionManager();
> >         manager.getParams().setStaleCheckingEnabled(true);
> >         manager.closeIdleConnections(1000L);
> >         manager.getParams().setConnectionTimeout(
> >                 (int) DEFAULT_CONNECTION_TIMEOUT);
> >
> >         httpclient = new HttpClient(manager);
> >         httpclient.getParams().setCookiePolicy(
> >                 CookiePolicy.BROWSER_COMPATIBILITY);
> >
> > However when I run this setup under load, I keep getting socket 
> > exceptions on the client side. I am using tomcat web server and I have
> 
> > increased the max-processor & access-count. I don't see any errors or 
> > exception in the tomcat log.
> >
> > However I keep getting a lot of these client socket bind exceptions 
> > after sometime. I guess it is happening due to the unavailability of 
> > ports. I would like to know some best practices in using the 
> > HttpClient under load. I am sure tha it is my bad, but wanted to get 
> > some input from the user/dev community. Should I have a pool of 
> > HttpClient objects and reuse them? Should I use the multi-threaded 
> > connection manager? I need to run this under heavier load and need to 
> > fix this soon. Any help, code snippets, or guidance would be greatly 
> > appreciated. Please let me know if you need any other information.
> >
> > Thanks in advance
> > Prasad
> >
> >
> > Client StackTrace:
> >
> > [ERROR] 01 Feb 2005 20:43:56,177 NDC[] T[Thread-20] HTTP          HTTP
> > I/O transport exception occurred
> > java.net.BindException: Address already in use: connect
> > 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> > 	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> > 	at
> > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> > 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> > 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364)
> > 	at java.net.Socket.connect(Socket.java:507)
> > 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> > 	at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > o
> > r
> > Impl.java:25)
> > 	at java.lang.reflect.Method.invoke(Method.java:585)
> > 	at
> > org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createS
> > o
> > c
> > ket(ReflectionSocketFactory.java:128)
> > 	at
> > org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.cr
> > e
> > a
> > teSocket(DefaultProtocolSocketFactory.java:124)
> > 	at
> > org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:
> 
> > 70
> > 5)
> > 	at
> > org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Http
> > M
> > e
> > thodDirector.java:359)
> > 	at
> > org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMet
> > h
> > o
> > dDirector.java:168)
> > 	at
> >
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java: 
> > 3
> > 93)
> > 	at
> >
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java: 
> > 3
> > 24)
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:  
> > httpclient-user-help@jakarta.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-user-help@jakarta.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-user-help@jakarta.apache.org


Mime
View raw message