hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: HttpClient IO hits the same core
Date Tue, 15 Oct 2013 10:06:54 GMT
On Tue, 2013-10-15 at 10:41 +0100, Ke Ren wrote:
> @Ahmed, don't have a dedicated server to test.
> @Oleg, yes, that's my confusion. I changed to http client 4.3 and still get
> the same result. Here is my test code. Can you see if I used anything
> wrongly? It's in scala but very similar to Java
> 
> import concurrent.{ExecutionContext, Await, Future, Promise}
> import scala.concurrent.duration._
> import java.util.concurrent.Executors
> import org.apache.http.impl.client.HttpClients
> import org.apache.http.client.methods.HttpGet
> import org.apache.http.util.EntityUtils
> 
> object MainTest {
>   def main(args: Array[String]) {
>     var num = 100
>     var jobs = 1000
>     if (args.size == 2) {
>       num = args(0).toInt
>       jobs = args(1).toInt
>     }
>     val executorService = Executors.newCachedThreadPool()
>     implicit val ec = ExecutionContext.fromExecutorService(executorService)
> 
>     var workers = List.empty[Worker]
>     var futures = List.empty[Future[Boolean]]
>     for (i <- 0 to num) {
>       val p = Promise[Boolean]
>       workers = workers.::(new Worker(p, jobs))
>       futures = futures.::(p.future)
>     }
> 
>     val allF = futures.tail.foldLeft(futures.head) {
>       case (future, rowFuture) => future.flatMap {
>         status =>
>           rowFuture.map(finished => status && finished)
>       }
>     }
> 
>     workers foreach {
>       worker =>
>         worker.start()
>     }
> 
>     Await.result(allF, 1 hour)
>     println("finished")
>   }
> }
> 
> class Worker(p: Promise[Boolean], jobs: Int) extends Thread {
>   override def run {
>     for (i <- 0 to jobs) {
>       try {
>         val httpclient = HttpClients.createDefault();
>         val httpget = new HttpGet("
> http://s3.amazonaws.com/panda.apelabs.net/gamecontent-unittest/Current-Config.json
> ");
>         val response = httpclient.execute(httpget);
>         try {
>           val entity = response.getEntity();
>           if (entity != null) {
>             EntityUtils.toByteArray(entity)
>           }
>         } finally {
>           response.close();
>           httpclient.close()
>         }
>       } catch {
>         case e: Throwable =>
>           e.printStackTrace()
>       }
>     }
>     p.success(true)
>   }
> }
> 
> 

You mentioned in your previous post you had been using one instance of
HttpClient shared by multiple threads. This is clearly not the case with
the code snippet above.

Oleg 

> On Tue, Oct 15, 2013 at 10:07 AM, Oleg Kalnichevski <olegk@apache.org>wrote:
> 
> > On Mon, 2013-10-14 at 22:22 +0100, Ke Ren wrote:
> > > Hi,
> > >
> > > We are sending massive http client requests on amazon ec2 instance with 8
> > > cores (m3.xlarge). We notice always one core is hammered the most (80%)
> > but
> > > others are just used around 30% to 40%. The httpclient we are using is
> > > 4.2.5. I reproduced this issue with very simple code on ec2 4 core
> > > instance. Basically I created one http client instance new
> > > DefaultHttpClient(mgr, params); mgr is a PoolingClientConnectionManager.
> > > This http client instance is shared by multiple threads. In this test,
> > each
> > > thread worker just gets the same 700k file from s3. The result is the
> > same.
> > > always just one core is hit heavily but very low usage on others. From
> > > jprofiler profiling, it looks the most cpu usage is on IO read from
> > > InputStream. I used EntityUtils.toByteArray to read data from entity.
> > From
> > > netstat, there are 50 connections. The thing confuses me is why it always
> > > hits the same core when http responses are processed from multiple
> > threads.
> > > Any ideas?
> > >
> > > Thanks in advance,
> > >
> > > Ke
> >
> > The only rational theory I can think of is that the ReentrantLock
> > guarding the connection pool favors certain threads due to 'unfair'
> > behavior used by default. Still, I would presume that threads should not
> > be bound to one core only and the load should get distributed evenly.
> >
> > Oleg
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message