commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhaohua Meng" <zm...@cgsh.com>
Subject Re: HttpClient: can it cause "too many open files" ?
Date Thu, 03 Nov 2005 15:04:27 GMT
Oleg,
You saved my day.
Thank you very much.

Zhaohua Meng
WebSphere Consultant



"Oleg Kalnichevski" <olegk@apache.org> 
11/03/2005 04:13 AM
Please respond to
"Jakarta Commons Users List" <commons-user@jakarta.apache.org>


To
commons-user@jakarta.apache.org
cc

Subject
Re: HttpClient: can it cause "too many open files" ?






On Wed, Nov 02, 2005 at 06:03:03PM -0500, Zhaohua Meng wrote:
> HttpClient gugus,
> 
> 
> I have a HttpClientEngine (see code listing at the bottom ) which wraps 
> commons HttpClient. It is used as following in my code deployed on SuSe 
> Linux.
> 
> for (int i=0; i<requests.length; i++)  }
>         HttpClientEngine engine = new HttpClientEngine ();
>         response = engine.doPost(requests[i]);
>         //code to process response
> }
> 
> In real situation t he request loop can be as big as 500+. After the 
code 
> running for a while I got "too many open files" exception. I know in 
Linux 
> a stream is considered a file. Please note that 
> "method.releaseConnection()" is called before doPost() returns a 
response 
> object. My questions are following:
> 
> 1. Will the following code work? In other words, can HttpClient object 
be 
> reused to post with different HttpBaseMethod?
> 
> HttpClientEngine engine = new HttpClientEngine ();
> for (int i=0; i<requests.length; i++)  }
>         response = engine.doPost(requests[i]);
>         //code to process response
> }
> 

Actually it is strongly recommended to reuse the HttpClient instance.
For details see the HttpClient optimization guide:

http://jakarta.apache.org/commons/httpclient/performance.html


> 2. Will my code leave some streams open thus causes "too many open 
files"? 
>

Yes it will. HttpMethod#releaseConnection() call does not close the
underlying socket if the connection can be kept alive. This method
merely returns connection back to the connection manager

> 
> 3. Besides "method.releaseConnection()", what other methods should I 
call 
> to close all streams?
> 

MultiThreadedHttpConnectionManager#shutdown()
SimpleHttpConnectionManager#dhutdown()


> Thanks,
> Zhaohua
> 
> 
> public class HttpClientEngine {
>         private HttpClient httpClient; //this is the commons HttpClient 
>         private HttpResponse response; //my proprietary class
>         private HttpRequest request; //my proprietary class
>         private URL url;
>         private Credentials credentials;
>         public HttpClientEngine() {
>                 httpClient = new HttpClient();
>         }
>         public HttpResponse doPost(HttpRequest request) {
>                 return 
> doPost(request,request.getURL(),request.getCredentials());
>         }
>         public HttpResponse doPost(HttpRequest request, URL url) {
>                 return doPost(request,url,request.getCredentials());
>         }
>         private static Logger logger = Logger.getInstance();
>         public HttpResponse doPost(HttpRequest request, URL url, 
> Credentials credentials) {
>                 this.request = request;
>                 this.url = url;
>                 this.credentials = credentials;
>                 return service(request,new PostMethod(url.toString()));
>         }
> 
>         public HttpResponse doGet(HttpRequest request) {
>                 return 
> doGet(request,request.getURL(),request.getCredentials());
>         }
>         public HttpResponse doGet(HttpRequest request, URL url) {
>                 return doGet(request,url,request.getCredentials());
>         }
>         public HttpResponse doGet(HttpRequest request, URL url, 
> Credentials credentials) {
>                 this.request = request;
>                 this.url = url;
>                 this.credentials = credentials;
>                 return service(request,new GetMethod(url.toString()));
>         }
> 
> 
>         private HttpResponse service(HttpRequest request, HttpMethodBase 

> method) {
>                 HttpHeader[] requestHeaders = request.getHeaders();
>                 if ( requestHeaders != null) {
>                         for (int i=0; i<requestHeaders.length; i++) {
>                                 //debug("header: 
> ["+requestHeaders[i].getName()+"]["+requestHeaders[i].getValue()+"]");
> 
> 
method.setRequestHeader(requestHeaders[i].getName(),requestHeaders[i].getValue());
>                         }
>                 }
>                 if (credentials != null) {
>                         UsernamePasswordCredentials login = 
>                                 new 
> 
UsernamePasswordCredentials(credentials.getName(),credentials.getPassword()); 

> 
>                         AuthScope auth=  new AuthScope(url.getHost(), 
> url.getPort(), null);
>                 httpClient.getState().setCredentials(auth, login);
>                 method.setDoAuthentication(true);
>                 }
>         if (method instanceof PostMethod) {
>                 try {
> ((PostMethod)method).setRequestEntity(new 
> ByteArrayRequestEntity(request.getBody()));
>                         } catch (Exception e) {
>                                 method.releaseConnection();
>                                 return null;
>                         }
>         }
>         try {
>                         int statusCode = 
httpClient.executeMethod(method);
>                         Header[] responseHeaders = 
> method.getResponseHeaders();
>                         HttpHeader[] headers = null;
>                         if (responseHeaders != null) {
>                                 headers = new 
> HttpHeader[responseHeaders.length];
>                                 for (int i=0; i<responseHeaders.length; 
> i++) {
>                                         headers[i] = new HttpHeader();
>  headers[i].setName(responseHeaders[i].getName());
>  headers[i].setValue(responseHeaders[i].getValue());
>                                 }
>                         }
> 
>                         response = 
> handler.process(method.getResponseBody(),statusCode, headers);
>                 } catch (HttpException e) {
>                         throw new RuntimeException(e.getMessage());
>                 } catch (IOException e) {
>                         throw new RuntimeException(e.getMessage());
>                 } finally {
>                         method.releaseConnection();
>         }
>                 return response;
>         }
>         private HttpResponseHandler handler;
>         public void setHttpResponseHandler(HttpResponseHandler handler) 
{
>                 this.handler = handler;
>         }
>         public static void debug(Object obj) {
> 
System.out.println(HttpClientEngine.class.getName()+"::"+obj.toString());
>         }
> 
> Zhaohua Meng
> WebSphere Consultant
> This message is being sent from a law firm and may contain confidential 
or privileged information.  If you are not the intended recipient, please 
advise the sender immediately by reply e-mail and delete this message and 
any attachments without retaining a copy.

---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org




This message is being sent from a law firm and may contain confidential or privileged information.
 If you are not the intended recipient, please advise the sender immediately by reply e-mail
and delete this message and any attachments without retaining a copy.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message