hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bit1129@163.com" <bit1...@163.com>
Subject Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3
Date Wed, 08 Jul 2015 06:49:14 GMT
Thanks Alexey for your suggestion. 
I have created a maven project and it is here : https://github.com/bit1129/bit-repo/tree/master/HttpClientUsage


Thanks!



bit1129@163.com
 
From: Alexey Panchenko
Date: 2015-07-08 14:05
To: HttpClient User Discussion
Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3
I would suggest going further and creating 2 buildable projects with maven
poms. So one can easily run it.
On Jul 8, 2015 11:57 AM, "bit1129@163.com" <bit1129@163.com> wrote:
 
> Ok, thanks Stefan!
>
> The github repo is https://github.com/bit1129/bit-repo . That are 5 java
> files there with the discription (Http Client 3.x and Http Client 4.3.6
> problematic code)
>
> Thanks very much for you guys's time on this!!
>
>
>
>
>
> bit1129@163.com
>
> From: Stefan Magnus Landrø
> Date: 2015-07-08 12:58
> To: HttpClient User Discussion
> Subject: Re: httpclient4 is extremely slow than correpsoning code of
> HttpClient3
> Can you create a github repo with your code? Reviewing gets easier then
>
> Sendt fra min iPhone
>
> > Den 8. jul. 2015 kl. 06.03 skrev "bit1129@163.com" <bit1129@163.com>:
> >
> > Thanks Alexey for the apply。 I tried it, and sorry that it still doesn't
> work. I am using one thread to sending http request for 1000 times every 20
> milliseconds. Some response time are as follows:
> > As you can see, the response time is not stable at all, (there are
> ocurrences with 7~10 seconds). If I don't use pool connection manager, then
> the response time are very stable, about 200 milliseconds for each request.
> >
> > Is there code example that issuing HTTP request with pool connection
> manager and apply the best practices for the CloseableHttpClient/response
> release/close or related things. So that, I can compare it with mine, and
> hopefully find whether  the problem is.
> > Thanks.
> >
> > My testing code snippet:
> >
> > public void test() {
> >        long allStart = System.currentTimeMillis();
> > while (i++ < loop) {
> > long start = System.currentTimeMillis();
> > HttpUtils.httpInvoke(TEST_URL, null, null);  //This the core code that
> issuing http request.
> > long a = System.currentTimeMillis() - start;
> > String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
> > writeToFile(timeSpent + "\n", logFile);
> > Thread.sleep(loopInterval);
> > }
> > writeToFile("Total time spent: " + (System.currentTimeMillis() -
> allStart) + "\n", logFile);
> >
> >
> > }
> >
> > 26
> > 30
> > 235
> > 439
> > 851
> > 1667
> > 1040
> > 234
> > 439
> > 851
> > 1671
> > 3310
> > 25
> > 27
> > 26
> > 238
> > 25
> > 234
> > 435
> > 24
> > 235
> > 24
> > 26
> > 27
> > 25
> > 232
> > 27
> > 26
> > 37
> > 29
> > 28
> > 251
> > 472
> > 915
> > 1804
> > 3580
> > 7132
> > 10057
> > 28
> > 26
> > 253
> > 26
> > 25
> > 23
> > 243
> > 24
> > 24
> > 27
> > 26
> > 1025
> > 27
> > 29
> > 24
> > 23
> >
> >
> >
> >
> > bit1129@163.com
> >
> > From: Alexey Panchenko
> > Date: 2015-07-08 11:42
> > To: HttpClient User Discussion
> > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
> HttpClient3
> > AFAIK there is no need in creating a new HttpClient instance every time,
> it
> > should be created once and reused.
> >
> >> On Wed, Jul 8, 2015 at 9:29 AM, bit1129@163.com <bit1129@163.com>
> wrote:
> >>
> >> One more thing that I noticed is:
> >>
> >> When I constructing the CloseableHttpClient, response time gets back to
> >> normal if I don't pass the PoolingHttpClientConnectionManager object to
> the
> >> HttpClientBuilder
> >>
> >>  HttpClientBuilder httpClientBuilder = HttpClients.custom();
> >>
> >>  /*comment this code so that no PoolingHttpClientConnectionManager will
> >> be used!*/
> >>  //httpClientBuilder.setConnectionManager(cm);
> >>
> >>  //other configurations
> >>  return httpClientBuilder.build();
> >>
> >> Seems that there are something related with
> >> PoolingHttpClientConnectionManager that cause the request's response
> time
> >> is abnormally long.
> >>
> >> Here is my code about PoolingHttpClientConnectionManager :
> >>
> >> cm = new PoolingHttpClientConnectionManager();
> >> cm.setMaxTotal(1024);
> >> cm.setDefaultMaxPerRoute(128);
> >>
> >>
> >>
> >> bit1129@163.com
> >>
> >> From: bit1129@163.com
> >> Date: 2015-07-08 10:25
> >> To: httpclient-users
> >> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
> >> HttpClient3
> >> Thanks Stefan for the reply.
> >>
> >> Do you mean that I move the CloseableHttpClient out of the try block
> like
> >> following,
> >>
> >> public String execute(HttpGet httpGet) {
> >> String body = "";
> >> CloseableHttpClient httpclient = this.getHttpclient();
> >> try{
> >> CloseableHttpResponse response = httpclient.execute(httpGet);
> >> int status = response.getStatusLine().getStatusCode();
> >> ///other codes goes for consuming response
> >>
> >> Unfortunately, It still doesn't work for me, and I still see that the
> >> request time is abnormally long.
> >>
> >> One thing I notice, is that when I set the socket time out to be 10
> >> seconds, there are couple of requests throw exception due to socket time
> >> out. So I wonder what may cause socket time out issue. With
> httpclient3, I
> >> see no such problems, and the longest request time is no more that 2
> >> seconds.
> >>
> >>
> >>
> >>
> >>
> >>
> >> bit1129@163.com
> >> From: Stefan Magnus Landrø
> >> Date: 2015-07-08 00:33
> >> To: HttpClient User Discussion
> >> Subject: Re: httpclient4 is extremely slow than correpsoning code of
> >> HttpClient3
> >>>> try{
> >>>> CloseableHttpClient httpclient =
> >> Try with resources will close client
> >> Sendt fra min iPhone
> >>> Den 7. jul. 2015 kl. 16.59 skrev Todd <bit1129@163.com>:
> >>>
> >>> Thanks Stefan for the reply. Could you please point to me which code
> you
> >> are referring to?
> >>>
> >>>
> >>>
> >>> At 2015-07-07 22:49:28, "Stefan Magnus Landrø" <
> stefan.landro@gmail.com>
> >> wrote:
> >>>> You shouldn't close http client on every request, only response
> >>>>
> >>>> Sendt fra min iPhone
> >>>>
> >>>>> Den 7. jul. 2015 kl. 16.05 skrev "bit1129@163.com" <bit1129@163.com
> >:
> >>>>>
> >>>>>
> >>>>> Can someone kindly help me on this? Thanks a lot!
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> bit1129@163.com
> >>>>>
> >>>>> From: bit1129@163.com
> >>>>> Date: 2015-07-07 20:51
> >>>>> To: httpclient-users
> >>>>> Subject: httpclient4 is extremely slow than correpsoning code of
> >> HttpClient3
> >>>>>
> >>>>> I have following http client code(HttpUtils and
> HttpClientManager)with
> >> HttpClient 4.3.6, and a httpclient code in 3.1, and a test case as
> well. I
> >> observed that HttpClient 4.3.6 is very much slowly than HttpClient 3.1
> >> code. In HttpClient 4.3.6 version code, there are a lot
> >>>>> socket time out error(the socket time is 10 seconds).
> >>>>>
> >>>>> I pasted the related classes and code below. It is kind of long
code,
> >> but I still would ask you do me a favor to review the http client
> >> configuration that may cause the problem. Many Thanks in advance.
> >>>>>
> >>>>> Following codes include:
> >>>>> 1. HttpClient 4.3.6 code to issue HttpGet request
> >>>>> 2. HttpClient 3.1 code to issue HttpGet request
> >>>>> 3. Test Case that demontrate the problem.
> >>>>>
> >>>>>
> >>>>> ############################################Http Client 4.3.6 Code
> >> goes here############################################
> >>
> ///////////////////////////HttpUtils//////////////////////////////////////////
> >>>>> import java.util.ArrayList;
> >>>>> import java.util.List;
> >>>>> import java.util.Map;
> >>>>>
> >>>>> import org.apache.http.Consts;
> >>>>> import org.apache.http.NameValuePair;
> >>>>> import org.apache.http.client.config.RequestConfig;
> >>>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
> >>>>> import org.apache.http.client.methods.HttpGet;
> >>>>> import org.apache.http.client.methods.HttpPost;
> >>>>> import org.apache.http.message.BasicNameValuePair;
> >>>>> import org.slf4j.Logger;
> >>>>> import org.slf4j.LoggerFactory;
> >>>>>
> >>>>>
> >>>>>
> >>>>> public class HttpUtils {
> >>>>>
> >>>>> ///This is the util method that will be used to issue http request.
> >>>>> public static String httpInvoke(String httpUrl, Map<String, Object>
> >> parameters, RequestConfig config){
> >>>>>  HttpClientManager httpClientConnectionManager =
> >> HttpClientManager.getHttpClientConnectionManagerInstance();
> >>>>>  HttpGet httpGet = new HttpGet(httpUrl);
> >>>>>  if(config != null ){
> >>>>>         httpGet.setConfig(config);
> >>>>> }
> >>>>> String result = httpClientConnectionManager.execute(httpGet);
> >>>>> return result;
> >>>>> }
> >>>>> }
> >>
> //////////////////////////////////HttpClientManager/////////////////////////////////
> >>>>>
> >>>>> import java.nio.charset.CodingErrorAction;
> >>>>> import java.util.ArrayList;
> >>>>> import java.util.Collection;
> >>>>>
> >>>>> import org.apache.commons.httpclient.Header;
> >>>>> import org.apache.commons.httpclient.HttpStatus;
> >>>>> import org.apache.commons.lang3.StringUtils;
> >>>>> import org.apache.http.Consts;
> >>>>> import org.apache.http.HttpEntity;
> >>>>> import org.apache.http.HttpHost;
> >>>>> import org.apache.http.client.config.RequestConfig;
> >>>>> import org.apache.http.client.entity.GzipDecompressingEntity;
> >>>>> import org.apache.http.client.methods.CloseableHttpResponse;
> >>>>> import org.apache.http.client.methods.HttpGet;
> >>>>> import org.apache.http.client.methods.HttpPost;
> >>>>> import org.apache.http.config.ConnectionConfig;
> >>>>> import org.apache.http.config.MessageConstraints;
> >>>>> import org.apache.http.config.SocketConfig;
> >>>>> import org.apache.http.impl.client.CloseableHttpClient;
> >>>>> import org.apache.http.impl.client.HttpClientBuilder;
> >>>>> import org.apache.http.impl.client.HttpClients;
> >>>>> import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
> >>>>> import org.apache.http.message.BasicHeader;
> >>>>> import org.apache.http.params.CoreProtocolPNames;
> >>>>> import org.apache.http.util.EntityUtils;
> >>>>> import org.slf4j.Logger;
> >>>>> import org.slf4j.LoggerFactory;
> >>>>>
> >>>>>
> >>>>> ///Main class that encapsulate the Http Client
> >>>>> public class HttpClientManager {
> >>>>> private static Logger logger =
> >> LoggerFactory.getLogger(HttpClientManager.class);
> >>>>>
> >>>>> private static int defaultConnectionTimeout = 10*1000;  //connection
> >> timeout
> >>>>> private static int defaultSocketTimeout = 10*1000;  //socket time
out
> >>>>> private static int connectionRequestTimeout = 10*1000;  //connection
> >> request timeout
> >>>>>
> >>>>> private static int defaultMaxRouteConnections = 128;
> >>>>> private static int defaultMaxTotalConnections = 1024;
> >>>>>
> >>>>> private static int defaultMaxHeaderCount = 200;
> >>>>> private static int defaultMaxLineLength = 2000;
> >>>>>
> >>>>> private static String Charset = "utf-8";
> >>>>>
> >>>>> private String proxyHost = null;
> >>>>> private String proxyPort =null;
> >>>>>
> >>>>> private PoolingHttpClientConnectionManager cm;
> >>>>>
> >>>>> private final static HttpClientManager httpClientConnectionManager
=
> >> new HttpClientManager();
> >>>>>
> >>>>> private HttpClientManager() {
> >>>>> logger.info("HttpClientManager initial!");
> >>>>> cm = new PoolingHttpClientConnectionManager();
> >>>>> cm.setMaxTotal(defaultMaxTotalConnections);
> >>>>> cm.setDefaultMaxPerRoute(defaultMaxRouteConnections);
> >>>>> }
> >>>>>
> >>>>>
> >>>>> private CloseableHttpClient getHttpclient(){
> >>>>> HttpClientBuilder httpClientBuilder = HttpClients.custom();
> >>>>> httpClientBuilder.setConnectionManager(cm);
> >>>>>
> >>>>> RequestConfig.Builder requestConfigBuilder = RequestConfig.custom()
> >>>>> .setConnectTimeout(defaultConnectionTimeout)
> >>>>> .setSocketTimeout(defaultSocketTimeout)
> >>>>> .setConnectionRequestTimeout(connectionRequestTimeout)
> >>>>> .setExpectContinueEnabled(false)
> >>>>> .setStaleConnectionCheckEnabled(true);
> >>>>>
> >>>>> if(StringUtils.isNotBlank(proxyHost) &&
> >> StringUtils.isNotBlank(proxyPort)){
> >>>>> try{
> >>>>> logger.info("using proxy, proxyHost:{}, proxyPort:{}", proxyHost,
> >> proxyPort);
> >>>>> int proxyPortInt = Integer.parseInt(proxyPort);
> >>>>> requestConfigBuilder.setProxy(new HttpHost(proxyHost, proxyPortInt));
> >>>>> } catch(Exception e){
> >>>>> logger.error("parseInt proxyPort:{}", proxyPort, e);
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> SocketConfig socketConfig =
> >> SocketConfig.custom().setTcpNoDelay(true).build();
> >>>>>
> >>>>> MessageConstraints messageConstraints =
> >>
> MessageConstraints.custom().setMaxHeaderCount(defaultMaxHeaderCount).setMaxLineLength(defaultMaxLineLength).build();
> >>>>>
> >>>>> ConnectionConfig connectionConfig = ConnectionConfig.custom()
> >>>>> .setMalformedInputAction(CodingErrorAction.IGNORE)
> >>>>> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> >>>>> .setCharset(Consts.UTF_8)
> >>>>> .setMessageConstraints(messageConstraints).build();
> >>>>>
> >>>>> Collection<BasicHeader> collection = new ArrayList<BasicHeader>();
> >>>>> collection.add(new BasicHeader("User-Agent", "Mozilla/5.0 (Windows;
> U;
> >> Windows NT 5.1; zh-CN; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3"));
> >>>>> collection.add(new BasicHeader("Accept-Language",
> >> "zh-cn,zh,en-US,en;q=0.5"));
> >>>>> collection.add(new BasicHeader("Accept-Charset", Charset));
> >>>>> collection.add(new BasicHeader("Accept-Encoding", "gzip"));
> >> httpClientBuilder.setDefaultRequestConfig(requestConfigBuilder.build());
> >>>>> httpClientBuilder.setDefaultSocketConfig(socketConfig);
> >>>>> httpClientBuilder.setDefaultConnectionConfig(connectionConfig);
> >>>>> httpClientBuilder.setDefaultHeaders(collection);
> >>>>>
> >>>>> return httpClientBuilder.build();
> >>>>> }
> >>>>>
> >>>>> ///This is the method that will be call the execute the HttpGet
> request
> >>>>> public String execute(HttpGet httpGet) {
> >>>>> String body = "";
> >>>>> try{
> >>>>> CloseableHttpClient httpclient = this.getHttpclient();
> >>>>> CloseableHttpResponse response = httpclient.execute(httpGet);
> >>>>> int status = response.getStatusLine().getStatusCode();
> >>>>> try {
> >>>>> if (status == HttpStatus.SC_OK) {
> >>>>> HttpEntity entity = response.getEntity();
> >>>>> if (entity != null) {
> >>>>> Header header = (Header) entity.getContentEncoding();
> >>>>> if(header != null && "gzip".equals(header.getValue())){
> >>>>> body = EntityUtils.toString(new GzipDecompressingEntity(entity),
> >> Charset);
> >>>>> } else {
> >>>>> body = EntityUtils.toString(entity, Charset);
> >>>>> }
> >>>>> }
> >>>>> } else {
> >>>>> logger.error("[httpClientManager] [fail] [httpGet:{}] [status:{}]",
> >> httpGet, status);
> >>>>> }
> >>>>> } finally {
> >>>>> response.close();
> >>>>> }
> >>>>> } catch(Exception e) {
> >>>>> logger.error("[module:httpClientManager] [action:execute]
> [httpGet:{}]
> >> [error:{}] ", httpGet, e.getMessage(), e);
> >>>>> }
> >>>>> return body;
> >>>>> }
> >>>>>
> >>>>>
> >>>>> ////Singleton object that will be used to access
> >> httpClientConnectionManager
> >>>>> public static HttpClientManager
> >> getHttpClientConnectionManagerInstance(){
> >>>>> return httpClientConnectionManager;
> >>>>> }
> >>>>>
> >>>>> }
> >>>>>
> >>>>>
> >>>>> ##################################################My HttpClient
1.3
> >> code goes here###########################################
> >>>>>
> >>>>> import org.apache.commons.httpclient.HttpClient;
> >>>>> import org.apache.commons.httpclient.HttpMethod;
> >>>>> import org.apache.commons.httpclient.methods.GetMethod;
> >>>>>
> >>>>> import java.io.BufferedReader;
> >>>>> import java.io.IOException;
> >>>>> import java.io.InputStream;
> >>>>> import java.io.InputStreamReader;
> >>>>>
> >>>>> public class HttpClientUtil {
> >>>>>
> >>>>> private static final HttpClientUtil INSTANCE = new HttpClientUtil();
> >>>>>
> >>>>> private static final int TIMEOUT = 10 * 1000; //10s
> >>>>>
> >>>>> private HttpClientUtil() {
> >>>>> }
> >>>>>
> >>>>> ///Actually, this class is more like a static class than Singleton
> >>>>> public static HttpClientUtil getInstance() {
> >>>>> return INSTANCE;
> >>>>> }
> >>>>>
> >>>>> //brand new http client per request
> >>>>> private HttpClient newHttpClient() {
> >>>>> HttpClient client = new HttpClient();
> >>
> client.getHttpConnectionManager().getParams().setConnectionTimeout(TIMEOUT);
> >>>>> client.getHttpConnectionManager().getParams().setSoTimeout(TIMEOUT);
> >>>>> return client;
> >>>>> }
> >>>>>
> >>>>> //FIXME The encoding should be provided when convert the response
> >> binary stream into string
> >>>>> public static String responseBodyAsString(HttpMethod method) throws
> >> IOException {
> >>>>> BufferedReader br = null;
> >>>>> String lsr = System.getProperty("line.separator");
> >>>>> try {
> >>>>> InputStream in = method.getResponseBodyAsStream();
> >>>>> br = new BufferedReader(new InputStreamReader(in, "UTF-8"));
> >>>>> StringBuffer sb = new StringBuffer();
> >>>>> String line;
> >>>>> while ((line = br.readLine()) != null) {
> >>>>> sb.append(line).append(lsr);
> >>>>> }
> >>>>> return sb.toString();
> >>>>> } finally {
> >>>>> if (br != null) {
> >>>>> br.close();
> >>>>> }
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> /////This is the method that will HttpGet to the url
> >>>>> public String get(String url) throws IOException {
> >>>>> GetMethod pm = new GetMethod(url);
> >>>>> pm.setRequestHeader("Connection", "close");
> >>>>> HttpClient client = newHttpClient();
> >>>>> try {
> >>>>> client.executeMethod(pm);
> >>>>> String response = responseBodyAsString(pm);
> >>>>> return response;
> >>>>> } finally {
> >>>>> pm.releaseConnection();
> >>>>> }
> >>>>> }
> >>>>> }
> >>>>>
> >>>>> ##########################################Test
> >> Case##########################################
> >>>>> The following code runs in the one thread in the junit test
> >>>>>      while (i++ < 5000) {
> >>>>> long start = System.currentTimeMillis();
> >>>>> HttpUtils.httpInvoke(TEST_URL, null, null);
> >>>>> long a = System.currentTimeMillis() - start;
> >>>>> String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
> >>>>> writeToFile(timeSpent + "\n", logFile);
> >>>>> Thread.sleep(loopInterval);
> >>>>> }
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> bit1129@163.com
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> >>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message