hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bit1129@163.com" <bit1...@163.com>
Subject Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3
Date Wed, 08 Jul 2015 05:57:08 GMT
Ok, thanks Stefan!

The github repo is https://github.com/bit1129/bit-repo . That are 5 java files there with
the discription (Http Client 3.x and Http Client 4.3.6 problematic code)

Thanks very much for you guys's time on this!!





bit1129@163.com
 
From: Stefan Magnus Landrø
Date: 2015-07-08 12:58
To: HttpClient User Discussion
Subject: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3
Can you create a github repo with your code? Reviewing gets easier then
 
Sendt fra min iPhone
 
> Den 8. jul. 2015 kl. 06.03 skrev "bit1129@163.com" <bit1129@163.com>:
> 
> Thanks Alexey for the apply。 I tried it, and sorry that it still doesn't work. I am
using one thread to sending http request for 1000 times every 20 milliseconds. Some response
time are as follows:
> As you can see, the response time is not stable at all, (there are ocurrences with 7~10
seconds). If I don't use pool connection manager, then the response time are very stable,
about 200 milliseconds for each request.
> 
> Is there code example that issuing HTTP request with pool connection manager and apply
the best practices for the CloseableHttpClient/response release/close or related things. So
that, I can compare it with mine, and hopefully find whether  the problem is.
> Thanks.
> 
> My testing code snippet:
> 
> public void test() {
>        long allStart = System.currentTimeMillis(); 
> while (i++ < loop) { 
> long start = System.currentTimeMillis(); 
> HttpUtils.httpInvoke(TEST_URL, null, null);  //This the core code that issuing http request.
> long a = System.currentTimeMillis() - start; 
> String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a); 
> writeToFile(timeSpent + "\n", logFile); 
> Thread.sleep(loopInterval); 
> } 
> writeToFile("Total time spent: " + (System.currentTimeMillis() - allStart) + "\n", logFile);
> 
> 
> }
> 
> 26 
> 30 
> 235 
> 439 
> 851 
> 1667 
> 1040 
> 234 
> 439 
> 851 
> 1671 
> 3310 
> 25 
> 27 
> 26 
> 238 
> 25 
> 234 
> 435 
> 24 
> 235 
> 24 
> 26 
> 27 
> 25 
> 232 
> 27 
> 26 
> 37 
> 29 
> 28 
> 251 
> 472 
> 915 
> 1804 
> 3580 
> 7132 
> 10057 
> 28 
> 26 
> 253 
> 26 
> 25 
> 23 
> 243 
> 24 
> 24 
> 27 
> 26 
> 1025 
> 27 
> 29 
> 24 
> 23
> 
> 
> 
> 
> bit1129@163.com
> 
> From: Alexey Panchenko
> Date: 2015-07-08 11:42
> To: HttpClient User Discussion
> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3
> AFAIK there is no need in creating a new HttpClient instance every time, it
> should be created once and reused.
> 
>> On Wed, Jul 8, 2015 at 9:29 AM, bit1129@163.com <bit1129@163.com> wrote:
>> 
>> One more thing that I noticed is:
>> 
>> When I constructing the CloseableHttpClient, response time gets back to
>> normal if I don't pass the PoolingHttpClientConnectionManager object to the
>> HttpClientBuilder
>> 
>>  HttpClientBuilder httpClientBuilder = HttpClients.custom();
>> 
>>  /*comment this code so that no PoolingHttpClientConnectionManager will
>> be used!*/
>>  //httpClientBuilder.setConnectionManager(cm);
>> 
>>  //other configurations
>>  return httpClientBuilder.build();
>> 
>> Seems that there are something related with
>> PoolingHttpClientConnectionManager that cause the request's response time
>> is abnormally long.
>> 
>> Here is my code about PoolingHttpClientConnectionManager :
>> 
>> cm = new PoolingHttpClientConnectionManager();
>> cm.setMaxTotal(1024);
>> cm.setDefaultMaxPerRoute(128);
>> 
>> 
>> 
>> bit1129@163.com
>> 
>> From: bit1129@163.com
>> Date: 2015-07-08 10:25
>> To: httpclient-users
>> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
>> HttpClient3
>> Thanks Stefan for the reply.
>> 
>> Do you mean that I move the CloseableHttpClient out of the try block like
>> following,
>> 
>> public String execute(HttpGet httpGet) {
>> String body = "";
>> CloseableHttpClient httpclient = this.getHttpclient();
>> try{
>> CloseableHttpResponse response = httpclient.execute(httpGet);
>> int status = response.getStatusLine().getStatusCode();
>> ///other codes goes for consuming response
>> 
>> Unfortunately, It still doesn't work for me, and I still see that the
>> request time is abnormally long.
>> 
>> One thing I notice, is that when I set the socket time out to be 10
>> seconds, there are couple of requests throw exception due to socket time
>> out. So I wonder what may cause socket time out issue. With httpclient3, I
>> see no such problems, and the longest request time is no more that 2
>> seconds.
>> 
>> 
>> 
>> 
>> 
>> 
>> bit1129@163.com
>> From: Stefan Magnus Landrø
>> Date: 2015-07-08 00:33
>> To: HttpClient User Discussion
>> Subject: Re: httpclient4 is extremely slow than correpsoning code of
>> HttpClient3
>>>> try{
>>>> CloseableHttpClient httpclient =
>> Try with resources will close client
>> Sendt fra min iPhone
>>> Den 7. jul. 2015 kl. 16.59 skrev Todd <bit1129@163.com>:
>>> 
>>> Thanks Stefan for the reply. Could you please point to me which code you
>> are referring to?
>>> 
>>> 
>>> 
>>> At 2015-07-07 22:49:28, "Stefan Magnus Landrø" <stefan.landro@gmail.com>
>> wrote:
>>>> You shouldn't close http client on every request, only response
>>>> 
>>>> Sendt fra min iPhone
>>>> 
>>>>> Den 7. jul. 2015 kl. 16.05 skrev "bit1129@163.com" <bit1129@163.com>:
>>>>> 
>>>>> 
>>>>> Can someone kindly help me on this? Thanks a lot!
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> bit1129@163.com
>>>>> 
>>>>> From: bit1129@163.com
>>>>> Date: 2015-07-07 20:51
>>>>> To: httpclient-users
>>>>> Subject: httpclient4 is extremely slow than correpsoning code of
>> HttpClient3
>>>>> 
>>>>> I have following http client code(HttpUtils and HttpClientManager)with
>> HttpClient 4.3.6, and a httpclient code in 3.1, and a test case as well. I
>> observed that HttpClient 4.3.6 is very much slowly than HttpClient 3.1
>> code. In HttpClient 4.3.6 version code, there are a lot
>>>>> socket time out error(the socket time is 10 seconds).
>>>>> 
>>>>> I pasted the related classes and code below. It is kind of long code,
>> but I still would ask you do me a favor to review the http client
>> configuration that may cause the problem. Many Thanks in advance.
>>>>> 
>>>>> Following codes include:
>>>>> 1. HttpClient 4.3.6 code to issue HttpGet request
>>>>> 2. HttpClient 3.1 code to issue HttpGet request
>>>>> 3. Test Case that demontrate the problem.
>>>>> 
>>>>> 
>>>>> ############################################Http Client 4.3.6 Code
>> goes here############################################
>> ///////////////////////////HttpUtils//////////////////////////////////////////
>>>>> import java.util.ArrayList;
>>>>> import java.util.List;
>>>>> import java.util.Map;
>>>>> 
>>>>> import org.apache.http.Consts;
>>>>> import org.apache.http.NameValuePair;
>>>>> import org.apache.http.client.config.RequestConfig;
>>>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
>>>>> import org.apache.http.client.methods.HttpGet;
>>>>> import org.apache.http.client.methods.HttpPost;
>>>>> import org.apache.http.message.BasicNameValuePair;
>>>>> import org.slf4j.Logger;
>>>>> import org.slf4j.LoggerFactory;
>>>>> 
>>>>> 
>>>>> 
>>>>> public class HttpUtils {
>>>>> 
>>>>> ///This is the util method that will be used to issue http request.
>>>>> public static String httpInvoke(String httpUrl, Map<String, Object>
>> parameters, RequestConfig config){
>>>>>  HttpClientManager httpClientConnectionManager =
>> HttpClientManager.getHttpClientConnectionManagerInstance();
>>>>>  HttpGet httpGet = new HttpGet(httpUrl);
>>>>>  if(config != null ){
>>>>>         httpGet.setConfig(config);
>>>>> }
>>>>> String result = httpClientConnectionManager.execute(httpGet);
>>>>> return result;
>>>>> }
>>>>> }
>> //////////////////////////////////HttpClientManager/////////////////////////////////
>>>>> 
>>>>> import java.nio.charset.CodingErrorAction;
>>>>> import java.util.ArrayList;
>>>>> import java.util.Collection;
>>>>> 
>>>>> import org.apache.commons.httpclient.Header;
>>>>> import org.apache.commons.httpclient.HttpStatus;
>>>>> import org.apache.commons.lang3.StringUtils;
>>>>> import org.apache.http.Consts;
>>>>> import org.apache.http.HttpEntity;
>>>>> import org.apache.http.HttpHost;
>>>>> import org.apache.http.client.config.RequestConfig;
>>>>> import org.apache.http.client.entity.GzipDecompressingEntity;
>>>>> import org.apache.http.client.methods.CloseableHttpResponse;
>>>>> import org.apache.http.client.methods.HttpGet;
>>>>> import org.apache.http.client.methods.HttpPost;
>>>>> import org.apache.http.config.ConnectionConfig;
>>>>> import org.apache.http.config.MessageConstraints;
>>>>> import org.apache.http.config.SocketConfig;
>>>>> import org.apache.http.impl.client.CloseableHttpClient;
>>>>> import org.apache.http.impl.client.HttpClientBuilder;
>>>>> import org.apache.http.impl.client.HttpClients;
>>>>> import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
>>>>> import org.apache.http.message.BasicHeader;
>>>>> import org.apache.http.params.CoreProtocolPNames;
>>>>> import org.apache.http.util.EntityUtils;
>>>>> import org.slf4j.Logger;
>>>>> import org.slf4j.LoggerFactory;
>>>>> 
>>>>> 
>>>>> ///Main class that encapsulate the Http Client
>>>>> public class HttpClientManager {
>>>>> private static Logger logger =
>> LoggerFactory.getLogger(HttpClientManager.class);
>>>>> 
>>>>> private static int defaultConnectionTimeout = 10*1000;  //connection
>> timeout
>>>>> private static int defaultSocketTimeout = 10*1000;  //socket time out
>>>>> private static int connectionRequestTimeout = 10*1000;  //connection
>> request timeout
>>>>> 
>>>>> private static int defaultMaxRouteConnections = 128;
>>>>> private static int defaultMaxTotalConnections = 1024;
>>>>> 
>>>>> private static int defaultMaxHeaderCount = 200;
>>>>> private static int defaultMaxLineLength = 2000;
>>>>> 
>>>>> private static String Charset = "utf-8";
>>>>> 
>>>>> private String proxyHost = null;
>>>>> private String proxyPort =null;
>>>>> 
>>>>> private PoolingHttpClientConnectionManager cm;
>>>>> 
>>>>> private final static HttpClientManager httpClientConnectionManager =
>> new HttpClientManager();
>>>>> 
>>>>> private HttpClientManager() {
>>>>> logger.info("HttpClientManager initial!");
>>>>> cm = new PoolingHttpClientConnectionManager();
>>>>> cm.setMaxTotal(defaultMaxTotalConnections);
>>>>> cm.setDefaultMaxPerRoute(defaultMaxRouteConnections);
>>>>> }
>>>>> 
>>>>> 
>>>>> private CloseableHttpClient getHttpclient(){
>>>>> HttpClientBuilder httpClientBuilder = HttpClients.custom();
>>>>> httpClientBuilder.setConnectionManager(cm);
>>>>> 
>>>>> RequestConfig.Builder requestConfigBuilder = RequestConfig.custom()
>>>>> .setConnectTimeout(defaultConnectionTimeout)
>>>>> .setSocketTimeout(defaultSocketTimeout)
>>>>> .setConnectionRequestTimeout(connectionRequestTimeout)
>>>>> .setExpectContinueEnabled(false)
>>>>> .setStaleConnectionCheckEnabled(true);
>>>>> 
>>>>> if(StringUtils.isNotBlank(proxyHost) &&
>> StringUtils.isNotBlank(proxyPort)){
>>>>> try{
>>>>> logger.info("using proxy, proxyHost:{}, proxyPort:{}", proxyHost,
>> proxyPort);
>>>>> int proxyPortInt = Integer.parseInt(proxyPort);
>>>>> requestConfigBuilder.setProxy(new HttpHost(proxyHost, proxyPortInt));
>>>>> } catch(Exception e){
>>>>> logger.error("parseInt proxyPort:{}", proxyPort, e);
>>>>> }
>>>>> }
>>>>> 
>>>>> SocketConfig socketConfig =
>> SocketConfig.custom().setTcpNoDelay(true).build();
>>>>> 
>>>>> MessageConstraints messageConstraints =
>> MessageConstraints.custom().setMaxHeaderCount(defaultMaxHeaderCount).setMaxLineLength(defaultMaxLineLength).build();
>>>>> 
>>>>> ConnectionConfig connectionConfig = ConnectionConfig.custom()
>>>>> .setMalformedInputAction(CodingErrorAction.IGNORE)
>>>>> .setUnmappableInputAction(CodingErrorAction.IGNORE)
>>>>> .setCharset(Consts.UTF_8)
>>>>> .setMessageConstraints(messageConstraints).build();
>>>>> 
>>>>> Collection<BasicHeader> collection = new ArrayList<BasicHeader>();
>>>>> collection.add(new BasicHeader("User-Agent", "Mozilla/5.0 (Windows; U;
>> Windows NT 5.1; zh-CN; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3"));
>>>>> collection.add(new BasicHeader("Accept-Language",
>> "zh-cn,zh,en-US,en;q=0.5"));
>>>>> collection.add(new BasicHeader("Accept-Charset", Charset));
>>>>> collection.add(new BasicHeader("Accept-Encoding", "gzip"));
>> httpClientBuilder.setDefaultRequestConfig(requestConfigBuilder.build());
>>>>> httpClientBuilder.setDefaultSocketConfig(socketConfig);
>>>>> httpClientBuilder.setDefaultConnectionConfig(connectionConfig);
>>>>> httpClientBuilder.setDefaultHeaders(collection);
>>>>> 
>>>>> return httpClientBuilder.build();
>>>>> }
>>>>> 
>>>>> ///This is the method that will be call the execute the HttpGet request
>>>>> public String execute(HttpGet httpGet) {
>>>>> String body = "";
>>>>> try{
>>>>> CloseableHttpClient httpclient = this.getHttpclient();
>>>>> CloseableHttpResponse response = httpclient.execute(httpGet);
>>>>> int status = response.getStatusLine().getStatusCode();
>>>>> try {
>>>>> if (status == HttpStatus.SC_OK) {
>>>>> HttpEntity entity = response.getEntity();
>>>>> if (entity != null) {
>>>>> Header header = (Header) entity.getContentEncoding();
>>>>> if(header != null && "gzip".equals(header.getValue())){
>>>>> body = EntityUtils.toString(new GzipDecompressingEntity(entity),
>> Charset);
>>>>> } else {
>>>>> body = EntityUtils.toString(entity, Charset);
>>>>> }
>>>>> }
>>>>> } else {
>>>>> logger.error("[httpClientManager] [fail] [httpGet:{}] [status:{}]",
>> httpGet, status);
>>>>> }
>>>>> } finally {
>>>>> response.close();
>>>>> }
>>>>> } catch(Exception e) {
>>>>> logger.error("[module:httpClientManager] [action:execute] [httpGet:{}]
>> [error:{}] ", httpGet, e.getMessage(), e);
>>>>> }
>>>>> return body;
>>>>> }
>>>>> 
>>>>> 
>>>>> ////Singleton object that will be used to access
>> httpClientConnectionManager
>>>>> public static HttpClientManager
>> getHttpClientConnectionManagerInstance(){
>>>>> return httpClientConnectionManager;
>>>>> }
>>>>> 
>>>>> }
>>>>> 
>>>>> 
>>>>> ##################################################My HttpClient 1.3
>> code goes here###########################################
>>>>> 
>>>>> import org.apache.commons.httpclient.HttpClient;
>>>>> import org.apache.commons.httpclient.HttpMethod;
>>>>> import org.apache.commons.httpclient.methods.GetMethod;
>>>>> 
>>>>> import java.io.BufferedReader;
>>>>> import java.io.IOException;
>>>>> import java.io.InputStream;
>>>>> import java.io.InputStreamReader;
>>>>> 
>>>>> public class HttpClientUtil {
>>>>> 
>>>>> private static final HttpClientUtil INSTANCE = new HttpClientUtil();
>>>>> 
>>>>> private static final int TIMEOUT = 10 * 1000; //10s
>>>>> 
>>>>> private HttpClientUtil() {
>>>>> }
>>>>> 
>>>>> ///Actually, this class is more like a static class than Singleton
>>>>> public static HttpClientUtil getInstance() {
>>>>> return INSTANCE;
>>>>> }
>>>>> 
>>>>> //brand new http client per request
>>>>> private HttpClient newHttpClient() {
>>>>> HttpClient client = new HttpClient();
>> client.getHttpConnectionManager().getParams().setConnectionTimeout(TIMEOUT);
>>>>> client.getHttpConnectionManager().getParams().setSoTimeout(TIMEOUT);
>>>>> return client;
>>>>> }
>>>>> 
>>>>> //FIXME The encoding should be provided when convert the response
>> binary stream into string
>>>>> public static String responseBodyAsString(HttpMethod method) throws
>> IOException {
>>>>> BufferedReader br = null;
>>>>> String lsr = System.getProperty("line.separator");
>>>>> try {
>>>>> InputStream in = method.getResponseBodyAsStream();
>>>>> br = new BufferedReader(new InputStreamReader(in, "UTF-8"));
>>>>> StringBuffer sb = new StringBuffer();
>>>>> String line;
>>>>> while ((line = br.readLine()) != null) {
>>>>> sb.append(line).append(lsr);
>>>>> }
>>>>> return sb.toString();
>>>>> } finally {
>>>>> if (br != null) {
>>>>> br.close();
>>>>> }
>>>>> }
>>>>> }
>>>>> 
>>>>> /////This is the method that will HttpGet to the url
>>>>> public String get(String url) throws IOException {
>>>>> GetMethod pm = new GetMethod(url);
>>>>> pm.setRequestHeader("Connection", "close");
>>>>> HttpClient client = newHttpClient();
>>>>> try {
>>>>> client.executeMethod(pm);
>>>>> String response = responseBodyAsString(pm);
>>>>> return response;
>>>>> } finally {
>>>>> pm.releaseConnection();
>>>>> }
>>>>> }
>>>>> }
>>>>> 
>>>>> ##########################################Test
>> Case##########################################
>>>>> The following code runs in the one thread in the junit test
>>>>>      while (i++ < 5000) {
>>>>> long start = System.currentTimeMillis();
>>>>> HttpUtils.httpInvoke(TEST_URL, null, null);
>>>>> long a = System.currentTimeMillis() - start;
>>>>> String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
>>>>> writeToFile(timeSpent + "\n", logFile);
>>>>> Thread.sleep(loopInterval);
>>>>> }
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> bit1129@163.com
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>> 
 
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message