hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ke Ren ...@spaceapegames.com>
Subject Re: HttpClient IO hits the same core
Date Tue, 15 Oct 2013 09:41:36 GMT
@Ahmed, don't have a dedicated server to test.
@Oleg, yes, that's my confusion. I changed to http client 4.3 and still get
the same result. Here is my test code. Can you see if I used anything
wrongly? It's in scala but very similar to Java

import concurrent.{ExecutionContext, Await, Future, Promise}
import scala.concurrent.duration._
import java.util.concurrent.Executors
import org.apache.http.impl.client.HttpClients
import org.apache.http.client.methods.HttpGet
import org.apache.http.util.EntityUtils

object MainTest {
  def main(args: Array[String]) {
    var num = 100
    var jobs = 1000
    if (args.size == 2) {
      num = args(0).toInt
      jobs = args(1).toInt
    }
    val executorService = Executors.newCachedThreadPool()
    implicit val ec = ExecutionContext.fromExecutorService(executorService)

    var workers = List.empty[Worker]
    var futures = List.empty[Future[Boolean]]
    for (i <- 0 to num) {
      val p = Promise[Boolean]
      workers = workers.::(new Worker(p, jobs))
      futures = futures.::(p.future)
    }

    val allF = futures.tail.foldLeft(futures.head) {
      case (future, rowFuture) => future.flatMap {
        status =>
          rowFuture.map(finished => status && finished)
      }
    }

    workers foreach {
      worker =>
        worker.start()
    }

    Await.result(allF, 1 hour)
    println("finished")
  }
}

class Worker(p: Promise[Boolean], jobs: Int) extends Thread {
  override def run {
    for (i <- 0 to jobs) {
      try {
        val httpclient = HttpClients.createDefault();
        val httpget = new HttpGet("
http://s3.amazonaws.com/panda.apelabs.net/gamecontent-unittest/Current-Config.json
");
        val response = httpclient.execute(httpget);
        try {
          val entity = response.getEntity();
          if (entity != null) {
            EntityUtils.toByteArray(entity)
          }
        } finally {
          response.close();
          httpclient.close()
        }
      } catch {
        case e: Throwable =>
          e.printStackTrace()
      }
    }
    p.success(true)
  }
}


On Tue, Oct 15, 2013 at 10:07 AM, Oleg Kalnichevski <olegk@apache.org>wrote:

> On Mon, 2013-10-14 at 22:22 +0100, Ke Ren wrote:
> > Hi,
> >
> > We are sending massive http client requests on amazon ec2 instance with 8
> > cores (m3.xlarge). We notice always one core is hammered the most (80%)
> but
> > others are just used around 30% to 40%. The httpclient we are using is
> > 4.2.5. I reproduced this issue with very simple code on ec2 4 core
> > instance. Basically I created one http client instance new
> > DefaultHttpClient(mgr, params); mgr is a PoolingClientConnectionManager.
> > This http client instance is shared by multiple threads. In this test,
> each
> > thread worker just gets the same 700k file from s3. The result is the
> same.
> > always just one core is hit heavily but very low usage on others. From
> > jprofiler profiling, it looks the most cpu usage is on IO read from
> > InputStream. I used EntityUtils.toByteArray to read data from entity.
> From
> > netstat, there are 50 connections. The thing confuses me is why it always
> > hits the same core when http responses are processed from multiple
> threads.
> > Any ideas?
> >
> > Thanks in advance,
> >
> > Ke
>
> The only rational theory I can think of is that the ReentrantLock
> guarding the connection pool favors certain threads due to 'unfair'
> behavior used by default. Still, I would presume that threads should not
> be bound to one core only and the load should get distributed evenly.
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message