hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3
Date Thu, 09 Jul 2015 07:50:26 GMT
On Thu, 2015-07-09 at 11:56 +0800, bit1129@163.com wrote:
> I checked the access.log about the response time  in the server is normal(about 200ms-300ms
per request)
> 
> For now, I simply can't explain  two things
> 1. If I use the httpclient 3.x to request the particular url in the test code, then there
is no sockect time error(The response time makes sense, with very few response time larger
that 1 second). But when I use the httpclient4.3.6,then there is always socket time error(response
time is larger than 10 seconds) occurring from time to time during the test

Make sure that both versions of HC generate compatible requests. For
instance, HC 4.x makes use of content compression by default (which
often decreases performance), whereas HC 3.x does not.

> 2. If I don't set the PoolConnectionManager into the CloseableHttpClient object, then
the socket time error is gone, and the response time gets back to normal.
> 

Default socket configs are ignored if one assigns a custom connection
manager. Please see javadocs of the HttpClientBuilder. 

Oleg

> The problem is obvious here, with one thread to send request sequentially, this simple
test case reproduced what we observed in our production environment. But for now I really
have no idea where the problem hides...
>  
> 
> 
> 
> bit1129@163.com
>  
> From: Stefan Magnus Landrø
> Date: 2015-07-08 18:08
> To: HttpClient User Discussion
> Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of HttpClient3
> I'd say it's in your server. Check the access logs of your server, and add
> resonse time logging there if you don't already have it in there.
>  
> 2015-07-08 12:03 GMT+02:00 bit1129@163.com <bit1129@163.com>:
>  
> > Thank you Stefan. In our production envrionment, we saw the same
> > phenomenon.
> > In the production, out socket time out is set to 2 seconds,then we see a
> > lot socket time out error for the url used in the test code.  In the mean
> > while, we request to this url with curl every second, and it shows that the
> > response time is about 200-300 ms, even if there are some occurrences that
> > are a little more that 1 seconds.
> >
> > Not sure where the problem goes.
> >
> >
> >
> >
> > bit1129@163.com
> >
> > From: Stefan Magnus Landrø
> > Date: 2015-07-08 17:40
> > To: HttpClient User Discussion
> > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
> > HttpClient3
> > Have you looked at the response times (access logs) in your server? Your
> > client setup look ok now btw
> >
> > 2015-07-08 10:12 GMT+02:00 bit1129@163.com <bit1129@163.com>:
> >
> > > Thanks Stefan and Oleg.
> > > 1. I refined the code to use single CloseableHttpClient object in
> > > HttpClientManager class,  but unfortunately, the problem is still there.
> > > Some sample out is as follows:
> > > 2. As with the log in the test, it would be helpful to use log4j.  Since
> > > it doesn't affect the response time of requests, so I will leave it there
> > > as it is for now.
> > >
> > >
> > > As you can see, there are 7 seconds, 10 seconds request there. I never
> > > have this with HttpClientUtil#get. If I don't set the pool connection
> > > manager to the CloseableHttpClient , then the response time will make
> > sense.
> > >
> > >
> > > 449
> > > 256
> > > 26
> > > 30
> > > 25
> > > 250
> > > 468
> > > 907
> > > 1787(>1000)
> > > 3548(>1000)
> > > 7072(>1000)
> > > 10038(>1000)
> > > 263
> > > 25
> > > 25
> > > 25
> > > 248
> > > 24
> > > 25
> > > 26
> > > 31
> > > 246
> > > 26
> > > 244
> > > 456
> > > 26
> > > 26
> > > 25
> > > 26
> > > 237
> > > 25
> > > 238
> > > 446
> > > 27
> > > 25
> > > 234
> > > 28
> > >
> > >
> > >
> > >
> > >
> > >
> > > bit1129@163.com
> > >
> > > From: Stefan Magnus Landrø
> > > Date: 2015-07-08 15:29
> > > To: HttpClient User Discussion
> > > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code of
> > > HttpClient3
> > > Few things here:
> > >
> > > HttpClientPoolUtilsTest:
> > >
> > > why not use a logging framework instead (log4j, logback, commons logging
> > > etc) of creating log files manually?
> > > Why all this static code?
> > >
> > > HttpClientManager Line 122: this willl invoke getHttpclient that will
> > build
> > > a brand new client on every time. Most inefficient.
> > >
> > > You should instead use a singleton pattern:
> > >
> > > public class HttpUtil {
> > >
> > > private CloseableHttpClient client = // ... build it in constructor
> > >
> > > public String execute(String url) {
> > >      client.execute( ... )
> > > }
> > >
> > >
> > > }
> > >
> > > Maybe
> > > https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html
> > > is
> > > a better fit for you?
> > >
> > >
> > >
> > >
> > >
> > > 2015-07-08 8:49 GMT+02:00 bit1129@163.com <bit1129@163.com>:
> > >
> > > > Thanks Alexey for your suggestion.
> > > > I have created a maven project and it is here :
> > > > https://github.com/bit1129/bit-repo/tree/master/HttpClientUsage
> > > >
> > > > Thanks!
> > > >
> > > >
> > > >
> > > > bit1129@163.com
> > > >
> > > > From: Alexey Panchenko
> > > > Date: 2015-07-08 14:05
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Re: httpclient4 is extremely slow than correpsoning code
> > of
> > > > HttpClient3
> > > > I would suggest going further and creating 2 buildable projects with
> > > maven
> > > > poms. So one can easily run it.
> > > > On Jul 8, 2015 11:57 AM, "bit1129@163.com" <bit1129@163.com> wrote:
> > > >
> > > > > Ok, thanks Stefan!
> > > > >
> > > > > The github repo is https://github.com/bit1129/bit-repo . That are
5
> > > java
> > > > > files there with the discription (Http Client 3.x and Http Client
> > 4.3.6
> > > > > problematic code)
> > > > >
> > > > > Thanks very much for you guys's time on this!!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > bit1129@163.com
> > > > >
> > > > > From: Stefan Magnus Landrø
> > > > > Date: 2015-07-08 12:58
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: httpclient4 is extremely slow than correpsoning code
of
> > > > > HttpClient3
> > > > > Can you create a github repo with your code? Reviewing gets easier
> > then
> > > > >
> > > > > Sendt fra min iPhone
> > > > >
> > > > > > Den 8. jul. 2015 kl. 06.03 skrev "bit1129@163.com" <
> > bit1129@163.com
> > > >:
> > > > > >
> > > > > > Thanks Alexey for the apply。 I tried it, and sorry that it
still
> > > > doesn't
> > > > > work. I am using one thread to sending http request for 1000 times
> > > every
> > > > 20
> > > > > milliseconds. Some response time are as follows:
> > > > > > As you can see, the response time is not stable at all, (there
are
> > > > > ocurrences with 7~10 seconds). If I don't use pool connection
> > manager,
> > > > then
> > > > > the response time are very stable, about 200 milliseconds for each
> > > > request.
> > > > > >
> > > > > > Is there code example that issuing HTTP request with pool
> > connection
> > > > > manager and apply the best practices for the
> > > CloseableHttpClient/response
> > > > > release/close or related things. So that, I can compare it with mine,
> > > and
> > > > > hopefully find whether  the problem is.
> > > > > > Thanks.
> > > > > >
> > > > > > My testing code snippet:
> > > > > >
> > > > > > public void test() {
> > > > > >        long allStart = System.currentTimeMillis();
> > > > > > while (i++ < loop) {
> > > > > > long start = System.currentTimeMillis();
> > > > > > HttpUtils.httpInvoke(TEST_URL, null, null);  //This the core
code
> > > that
> > > > > issuing http request.
> > > > > > long a = System.currentTimeMillis() - start;
> > > > > > String timeSpent = "" + (a >= 1000 ? a + "(>1000)" : a);
> > > > > > writeToFile(timeSpent + "\n", logFile);
> > > > > > Thread.sleep(loopInterval);
> > > > > > }
> > > > > > writeToFile("Total time spent: " + (System.currentTimeMillis()
-
> > > > > allStart) + "\n", logFile);
> > > > > >
> > > > > >
> > > > > > }
> > > > > >
> > > > > > 26
> > > > > > 30
> > > > > > 235
> > > > > > 439
> > > > > > 851
> > > > > > 1667
> > > > > > 1040
> > > > > > 234
> > > > > > 439
> > > > > > 851
> > > > > > 1671
> > > > > > 3310
> > > > > > 25
> > > > > > 27
> > > > > > 26
> > > > > > 238
> > > > > > 25
> > > > > > 234
> > > > > > 435
> > > > > > 24
> > > > > > 235
> > > > > > 24
> > > > > > 26
> > > > > > 27
> > > > > > 25
> > > > > > 232
> > > > > > 27
> > > > > > 26
> > > > > > 37
> > > > > > 29
> > > > > > 28
> > > > > > 251
> > > > > > 472
> > > > > > 915
> > > > > > 1804
> > > > > > 3580
> > > > > > 7132
> > > > > > 10057
> > > > > > 28
> > > > > > 26
> > > > > > 253
> > > > > > 26
> > > > > > 25
> > > > > > 23
> > > > > > 243
> > > > > > 24
> > > > > > 24
> > > > > > 27
> > > > > > 26
> > > > > > 1025
> > > > > > 27
> > > > > > 29
> > > > > > 24
> > > > > > 23
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > bit1129@163.com
> > > > > >
> > > > > > From: Alexey Panchenko
> > > > > > Date: 2015-07-08 11:42
> > > > > > To: HttpClient User Discussion
> > > > > > Subject: Re: Re: httpclient4 is extremely slow than correpsoning
> > code
> > > > of
> > > > > HttpClient3
> > > > > > AFAIK there is no need in creating a new HttpClient instance
every
> > > > time,
> > > > > it
> > > > > > should be created once and reused.
> > > > > >
> > > > > >> On Wed, Jul 8, 2015 at 9:29 AM, bit1129@163.com <bit1129@163.com>
> > > > > wrote:
> > > > > >>
> > > > > >> One more thing that I noticed is:
> > > > > >>
> > > > > >> When I constructing the CloseableHttpClient, response time
gets
> > back
> > > > to
> > > > > >> normal if I don't pass the PoolingHttpClientConnectionManager
> > object
> > > > to
> > > > > the
> > > > > >> HttpClientBuilder
> > > > > >>
> > > > > >>  HttpClientBuilder httpClientBuilder = HttpClients.custom();
> > > > > >>
> > > > > >>  /*comment this code so that no PoolingHttpClientConnectionManager
> > > > will
> > > > > >> be used!*/
> > > > > >>  //httpClientBuilder.setConnectionManager(cm);
> > > > > >>
> > > > > >>  //other configurations
> > > > > >>  return httpClientBuilder.build();
> > > > > >>
> > > > > >> Seems that there are something related with
> > > > > >> PoolingHttpClientConnectionManager that cause the request's
> > response
> > > > > time
> > > > > >> is abnormally long.
> > > > > >>
> > > > > >> Here is my code about PoolingHttpClientConnectionManager
:
> > > > > >>
> > > > > >> cm = new PoolingHttpClientConnectionManager();
> > > > > >> cm.setMaxTotal(1024);
> > > > > >> cm.setDefaultMaxPerRoute(128);
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> bit1129@163.com
> > > > > >>
> > > > > >> From: bit1129@163.com
> > > > > >> Date: 2015-07-08 10:25
> > > > > >> To: httpclient-users
> > > > > >> Subject: Re: Re: httpclient4 is extremely slow than correpsoning
> > > code
> > > > of
> > > > > >> HttpClient3
> > > > > >> Thanks Stefan for the reply.
> > > > > >>
> > > > > >> Do you mean that I move the CloseableHttpClient out of the
try
> > block
> > > > > like
> > > > > >> following,
> > > > > >>
> > > > > >> public String execute(HttpGet httpGet) {
> > > > > >> String body = "";
> > > > > >> CloseableHttpClient httpclient = this.getHttpclient();
> > > > > >> try{
> > > > > >> CloseableHttpResponse response = httpclient.execute(httpGet);
> > > > > >> int status = response.getStatusLine().getStatusCode();
> > > > > >> ///other codes goes for consuming response
> > > > > >>
> > > > > >> Unfortunately, It still doesn't work for me, and I still
see that
> > > the
> > > > > >> request time is abnormally long.
> > > > > >>
> > > > > >> One thing I notice, is that when I set the socket time out
to be
> > 10
> > > > > >> seconds, there are couple of requests throw exception due
to
> > socket
> > > > time
> > > > > >> out. So I wonder what may cause socket time out issue. With
> > > > > httpclient3, I
> > > > > >> see no such problems, and the longest request time is no
more
> > that 2
> > > > > >> seconds.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> bit1129@163.com
> > > > > >> From: Stefan Magnus Landrø
> > > > > >> Date: 2015-07-08 00:33
> > > > > >> To: HttpClient User Discussion
> > > > > >> Subject: Re: httpclient4 is extremely slow than correpsoning
code
> > of
> > > > > >> HttpClient3
> > > > > >>>> try{
> > > > > >>>> CloseableHttpClient httpclient =
> > > > > >> Try with resources will close client
> > > > > >> Sendt fra min iPhone
> > > > > >>> Den 7. jul. 2015 kl. 16.59 skrev Todd <bit1129@163.com>:
> > > > > >>>
> > > > > >>> Thanks Stefan for the reply. Could you please point
to me which
> > > code
> > > > > you
> > > > > >> are referring to?
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> At 2015-07-07 22:49:28, "Stefan Magnus Landrø" <
> > > > > stefan.landro@gmail.com>
> > > > > >> wrote:
> > > > > >>>> You shouldn't close http client on every request,
only response
> > > > > >>>>
> > > > > >>>> Sendt fra min iPhone
> > > > > >>>>
> > > > > >>>>> Den 7. jul. 2015 kl. 16.05 skrev "bit1129@163.com"
<
> > > > bit1129@163.com
> > > > > >:
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> Can someone kindly help me on this? Thanks a
lot!
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> bit1129@163.com
> > > > > >>>>>
> > > > > >>>>> From: bit1129@163.com
> > > > > >>>>> Date: 2015-07-07 20:51
> > > > > >>>>> To: httpclient-users
> > > > > >>>>> Subject: httpclient4 is extremely slow than
correpsoning code
> > of
> > > > > >> HttpClient3
> > > > > >>>>>
> > > > > >>>>> I have following http client code(HttpUtils
and
> > > > > HttpClientManager)with
> > > > > >> HttpClient 4.3.6, and a httpclient code in 3.1, and a test
case as
> > > > > well. I
> > > > > >> observed that HttpClient 4.3.6 is very much slowly than
HttpClient
> > > 3.1
> > > > > >> code. In HttpClient 4.3.6 version code, there are a lot
> > > > > >>>>> socket time out error(the socket time is 10
seconds).
> > > > > >>>>>
> > > > > >>>>> I pasted the related classes and code below.
It is kind of long
> > > > code,
> > > > > >> but I still would ask you do me a favor to review the http
client
> > > > > >> configuration that may cause the problem. Many Thanks in
advance.
> > > > > >>>>>
> > > > > >>>>> Following codes include:
> > > > > >>>>> 1. HttpClient 4.3.6 code to issue HttpGet request
> > > > > >>>>> 2. HttpClient 3.1 code to issue HttpGet request
> > > > > >>>>> 3. Test Case that demontrate the problem.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> ############################################Http
Client 4.3.6
> > > Code
> > > > > >> goes here############################################
> > > > > >>
> > > > >
> > > >
> > >
> > ///////////////////////////HttpUtils//////////////////////////////////////////
> > > > > >>>>> import java.util.ArrayList;
> > > > > >>>>> import java.util.List;
> > > > > >>>>> import java.util.Map;
> > > > > >>>>>
> > > > > >>>>> import org.apache.http.Consts;
> > > > > >>>>> import org.apache.http.NameValuePair;
> > > > > >>>>> import org.apache.http.client.config.RequestConfig;
> > > > > >>>>> import org.apache.http.client.entity.UrlEncodedFormEntity;
> > > > > >>>>> import org.apache.http.client.methods.HttpGet;
> > > > > >>>>> import org.apache.http.client.methods.HttpPost;
> > > > > >>>>> import org.apache.http.message.BasicNameValuePair;
> > > > > >>>>> import org.slf4j.Logger;
> > > > > >>>>> import org.slf4j.LoggerFactory;
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> public class HttpUtils {
> > > > > >>>>>
> > > > > >>>>> ///This is the util method that will be used
to issue http
> > > request.
> > > > > >>>>> public static String httpInvoke(String httpUrl,
Map<String,
> > > Object>
> > > > > >> parameters, RequestConfig config){
> > > > > >>>>>  HttpClientManager httpClientConnectionManager
=
> > > > > >> HttpClientManager.getHttpClientConnectionManagerInstance();
> > > > > >>>>>  HttpGet httpGet = new HttpGet(httpUrl);
> > > > > >>>>>  if(config != null ){
> > > > > >>>>>         httpGet.setConfig(config);
> > > > > >>>>> }
> > > > > >>>>> String result = httpClientConnectionManager.execute(httpGet);
> > > > > >>>>> return result;
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>
> > > > >
> > > >
> > >
> > //////////////////////////////////HttpClientManager/////////////////////////////////
> > > > > >>>>>
> > > > > >>>>> import java.nio.charset.CodingErrorAction;
> > > > > >>>>> import java.util.ArrayList;
> > > > > >>>>> import java.util.Collection;
> > > > > >>>>>
> > > > > >>>>> import org.apache.commons.httpclient.Header;
> > > > > >>>>> import org.apache.commons.httpclient.HttpStatus;
> > > > > >>>>> import org.apache.commons.lang3.StringUtils;
> > > > > >>>>> import org.apache.http.Consts;
> > > > > >>>>> import org.apache.http.HttpEntity;
> > > > > >>>>> import org.apache.http.HttpHost;
> > > > > >>>>> import org.apache.http.client.config.RequestConfig;
> > > > > >>>>> import org.apache.http.client.entity.GzipDecompressingEntity;
> > > > > >>>>> import org.apache.http.client.methods.CloseableHttpResponse;
> > > > > >>>>> import org.apache.http.client.methods.HttpGet;
> > > > > >>>>> import org.apache.http.client.methods.HttpPost;
> > > > > >>>>> import org.apache.http.config.ConnectionConfig;
> > > > > >>>>> import org.apache.http.config.MessageConstraints;
> > > > > >>>>> import org.apache.http.config.SocketConfig;
> > > > > >>>>> import org.apache.http.impl.client.CloseableHttpClient;
> > > > > >>>>> import org.apache.http.impl.client.HttpClientBuilder;
> > > > > >>>>> import org.apache.http.impl.client.HttpClients;
> > > > > >>>>> import
> > > > org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
> > > > > >>>>> import org.apache.http.message.BasicHeader;
> > > > > >>>>> import org.apache.http.params.CoreProtocolPNames;
> > > > > >>>>> import org.apache.http.util.EntityUtils;
> > > > > >>>>> import org.slf4j.Logger;
> > > > > >>>>> import org.slf4j.LoggerFactory;
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> ///Main class that encapsulate the Http Client
> > > > > >>>>> public class HttpClientManager {
> > > > > >>>>> private static Logger logger =
> > > > > >> LoggerFactory.getLogger(HttpClientManager.class);
> > > > > >>>>>
> > > > > >>>>> private static int defaultConnectionTimeout
= 10*1000;
> > > > //connection
> > > > > >> timeout
> > > > > >>>>> private static int defaultSocketTimeout = 10*1000;
 //socket
> > time
> > > > out
> > > > > >>>>> private static int connectionRequestTimeout
= 10*1000;
> > > > //connection
> > > > > >> request timeout
> > > > > >>>>>
> > > > > >>>>> private static int defaultMaxRouteConnections
= 128;
> > > > > >>>>> private static int defaultMaxTotalConnections
= 1024;
> > > > > >>>>>
> > > > > >>>>> private static int defaultMaxHeaderCount = 200;
> > > > > >>>>> private static int defaultMaxLineLength = 2000;
> > > > > >>>>>
> > > > > >>>>> private static String Charset = "utf-8";
> > > > > >>>>>
> > > > > >>>>> private String proxyHost = null;
> > > > > >>>>> private String proxyPort =null;
> > > > > >>>>>
> > > > > >>>>> private PoolingHttpClientConnectionManager cm;
> > > > > >>>>>
> > > > > >>>>> private final static HttpClientManager
> > > httpClientConnectionManager
> > > > =
> > > > > >> new HttpClientManager();
> > > > > >>>>>
> > > > > >>>>> private HttpClientManager() {
> > > > > >>>>> logger.info("HttpClientManager initial!");
> > > > > >>>>> cm = new PoolingHttpClientConnectionManager();
> > > > > >>>>> cm.setMaxTotal(defaultMaxTotalConnections);
> > > > > >>>>> cm.setDefaultMaxPerRoute(defaultMaxRouteConnections);
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> private CloseableHttpClient getHttpclient(){
> > > > > >>>>> HttpClientBuilder httpClientBuilder = HttpClients.custom();
> > > > > >>>>> httpClientBuilder.setConnectionManager(cm);
> > > > > >>>>>
> > > > > >>>>> RequestConfig.Builder requestConfigBuilder =
> > > RequestConfig.custom()
> > > > > >>>>> .setConnectTimeout(defaultConnectionTimeout)
> > > > > >>>>> .setSocketTimeout(defaultSocketTimeout)
> > > > > >>>>> .setConnectionRequestTimeout(connectionRequestTimeout)
> > > > > >>>>> .setExpectContinueEnabled(false)
> > > > > >>>>> .setStaleConnectionCheckEnabled(true);
> > > > > >>>>>
> > > > > >>>>> if(StringUtils.isNotBlank(proxyHost) &&
> > > > > >> StringUtils.isNotBlank(proxyPort)){
> > > > > >>>>> try{
> > > > > >>>>> logger.info("using proxy, proxyHost:{}, proxyPort:{}",
> > > proxyHost,
> > > > > >> proxyPort);
> > > > > >>>>> int proxyPortInt = Integer.parseInt(proxyPort);
> > > > > >>>>> requestConfigBuilder.setProxy(new HttpHost(proxyHost,
> > > > proxyPortInt));
> > > > > >>>>> } catch(Exception e){
> > > > > >>>>> logger.error("parseInt proxyPort:{}", proxyPort,
e);
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> SocketConfig socketConfig =
> > > > > >> SocketConfig.custom().setTcpNoDelay(true).build();
> > > > > >>>>>
> > > > > >>>>> MessageConstraints messageConstraints =
> > > > > >>
> > > > >
> > > >
> > >
> > MessageConstraints.custom().setMaxHeaderCount(defaultMaxHeaderCount).setMaxLineLength(defaultMaxLineLength).build();
> > > > > >>>>>
> > > > > >>>>> ConnectionConfig connectionConfig = ConnectionConfig.custom()
> > > > > >>>>> .setMalformedInputAction(CodingErrorAction.IGNORE)
> > > > > >>>>> .setUnmappableInputAction(CodingErrorAction.IGNORE)
> > > > > >>>>> .setCharset(Consts.UTF_8)
> > > > > >>>>> .setMessageConstraints(messageConstraints).build();
> > > > > >>>>>
> > > > > >>>>> Collection<BasicHeader> collection = new
> > > ArrayList<BasicHeader>();
> > > > > >>>>> collection.add(new BasicHeader("User-Agent",
"Mozilla/5.0
> > > (Windows;
> > > > > U;
> > > > > >> Windows NT 5.1; zh-CN; rv:1.9.0.3) Gecko/2008092417
> > > Firefox/3.0.3"));
> > > > > >>>>> collection.add(new BasicHeader("Accept-Language",
> > > > > >> "zh-cn,zh,en-US,en;q=0.5"));
> > > > > >>>>> collection.add(new BasicHeader("Accept-Charset",
Charset));
> > > > > >>>>> collection.add(new BasicHeader("Accept-Encoding",
"gzip"));
> > > > > >>
> > > >
> > httpClientBuilder.setDefaultRequestConfig(requestConfigBuilder.build());
> > > > > >>>>> httpClientBuilder.setDefaultSocketConfig(socketConfig);
> > > > > >>>>> httpClientBuilder.setDefaultConnectionConfig(connectionConfig);
> > > > > >>>>> httpClientBuilder.setDefaultHeaders(collection);
> > > > > >>>>>
> > > > > >>>>> return httpClientBuilder.build();
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> ///This is the method that will be call the
execute the HttpGet
> > > > > request
> > > > > >>>>> public String execute(HttpGet httpGet) {
> > > > > >>>>> String body = "";
> > > > > >>>>> try{
> > > > > >>>>> CloseableHttpClient httpclient = this.getHttpclient();
> > > > > >>>>> CloseableHttpResponse response = httpclient.execute(httpGet);
> > > > > >>>>> int status = response.getStatusLine().getStatusCode();
> > > > > >>>>> try {
> > > > > >>>>> if (status == HttpStatus.SC_OK) {
> > > > > >>>>> HttpEntity entity = response.getEntity();
> > > > > >>>>> if (entity != null) {
> > > > > >>>>> Header header = (Header) entity.getContentEncoding();
> > > > > >>>>> if(header != null && "gzip".equals(header.getValue())){
> > > > > >>>>> body = EntityUtils.toString(new
> > GzipDecompressingEntity(entity),
> > > > > >> Charset);
> > > > > >>>>> } else {
> > > > > >>>>> body = EntityUtils.toString(entity, Charset);
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>>>> } else {
> > > > > >>>>> logger.error("[httpClientManager] [fail] [httpGet:{}]
> > > [status:{}]",
> > > > > >> httpGet, status);
> > > > > >>>>> }
> > > > > >>>>> } finally {
> > > > > >>>>> response.close();
> > > > > >>>>> }
> > > > > >>>>> } catch(Exception e) {
> > > > > >>>>> logger.error("[module:httpClientManager] [action:execute]
> > > > > [httpGet:{}]
> > > > > >> [error:{}] ", httpGet, e.getMessage(), e);
> > > > > >>>>> }
> > > > > >>>>> return body;
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> ////Singleton object that will be used to access
> > > > > >> httpClientConnectionManager
> > > > > >>>>> public static HttpClientManager
> > > > > >> getHttpClientConnectionManagerInstance(){
> > > > > >>>>> return httpClientConnectionManager;
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> ##################################################My
HttpClient
> > > 1.3
> > > > > >> code goes here###########################################
> > > > > >>>>>
> > > > > >>>>> import org.apache.commons.httpclient.HttpClient;
> > > > > >>>>> import org.apache.commons.httpclient.HttpMethod;
> > > > > >>>>> import org.apache.commons.httpclient.methods.GetMethod;
> > > > > >>>>>
> > > > > >>>>> import java.io.BufferedReader;
> > > > > >>>>> import java.io.IOException;
> > > > > >>>>> import java.io.InputStream;
> > > > > >>>>> import java.io.InputStreamReader;
> > > > > >>>>>
> > > > > >>>>> public class HttpClientUtil {
> > > > > >>>>>
> > > > > >>>>> private static final HttpClientUtil INSTANCE
= new
> > > > HttpClientUtil();
> > > > > >>>>>
> > > > > >>>>> private static final int TIMEOUT = 10 * 1000;
//10s
> > > > > >>>>>
> > > > > >>>>> private HttpClientUtil() {
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> ///Actually, this class is more like a static
class than
> > > Singleton
> > > > > >>>>> public static HttpClientUtil getInstance() {
> > > > > >>>>> return INSTANCE;
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> //brand new http client per request
> > > > > >>>>> private HttpClient newHttpClient() {
> > > > > >>>>> HttpClient client = new HttpClient();
> > > > > >>
> > > > >
> > > >
> > >
> > client.getHttpConnectionManager().getParams().setConnectionTimeout(TIMEOUT);
> > > > > >>>>>
> > > > client.getHttpConnectionManager().getParams().setSoTimeout(TIMEOUT);
> > > > > >>>>> return client;
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> //FIXME The encoding should be provided when
convert the
> > response
> > > > > >> binary stream into string
> > > > > >>>>> public static String responseBodyAsString(HttpMethod
method)
> > > throws
> > > > > >> IOException {
> > > > > >>>>> BufferedReader br = null;
> > > > > >>>>> String lsr = System.getProperty("line.separator");
> > > > > >>>>> try {
> > > > > >>>>> InputStream in = method.getResponseBodyAsStream();
> > > > > >>>>> br = new BufferedReader(new InputStreamReader(in,
"UTF-8"));
> > > > > >>>>> StringBuffer sb = new StringBuffer();
> > > > > >>>>> String line;
> > > > > >>>>> while ((line = br.readLine()) != null) {
> > > > > >>>>> sb.append(line).append(lsr);
> > > > > >>>>> }
> > > > > >>>>> return sb.toString();
> > > > > >>>>> } finally {
> > > > > >>>>> if (br != null) {
> > > > > >>>>> br.close();
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> /////This is the method that will HttpGet to
the url
> > > > > >>>>> public String get(String url) throws IOException
{
> > > > > >>>>> GetMethod pm = new GetMethod(url);
> > > > > >>>>> pm.setRequestHeader("Connection", "close");
> > > > > >>>>> HttpClient client = newHttpClient();
> > > > > >>>>> try {
> > > > > >>>>> client.executeMethod(pm);
> > > > > >>>>> String response = responseBodyAsString(pm);
> > > > > >>>>> return response;
> > > > > >>>>> } finally {
> > > > > >>>>> pm.releaseConnection();
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> ##########################################Test
> > > > > >> Case##########################################
> > > > > >>>>> The following code runs in the one thread in
the junit test
> > > > > >>>>>      while (i++ < 5000) {
> > > > > >>>>> long start = System.currentTimeMillis();
> > > > > >>>>> HttpUtils.httpInvoke(TEST_URL, null, null);
> > > > > >>>>> long a = System.currentTimeMillis() - start;
> > > > > >>>>> String timeSpent = "" + (a >= 1000 ? a +
"(>1000)" : a);
> > > > > >>>>> writeToFile(timeSpent + "\n", logFile);
> > > > > >>>>> Thread.sleep(loopInterval);
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> bit1129@163.com
> > > > > >>>>
> > > > > >>>>
> > > > ---------------------------------------------------------------------
> > > > > >>>> To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > > > >>>> For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > > > >>
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > BEKK Open
> > > http://open.bekk.no
> > >
> > > TesTcl - a unit test framework for iRules
> > > http://testcl.com
> > >
> >
> >
> >
> > --
> > BEKK Open
> > http://open.bekk.no
> >
> > TesTcl - a unit test framework for iRules
> > http://testcl.com
> >
>  
>  
>  
> -- 
> BEKK Open
> http://open.bekk.no
>  
> TesTcl - a unit test framework for iRules
> http://testcl.com



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message