hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: HTTPCore-NIO: Server rejecting socket connections?
Date Tue, 31 Jan 2012 14:18:37 GMT
On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
> On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> > Hi,
> > 
> > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <olegk@apache.org> wrote:
> > 
> > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > > > Hi,
> > > >
> > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <olegk@apache.org>
> > > wrote:
> > > >
> > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I have written an HTTP server using HTTPCore-NIO and currently
> > > testing it
> > > > > > with httpcore-ab [1]. I have been seeing very good TPS values
so
> > > far. But
> > > > > > when I try to send 5KB HTTP messages with 2500 and concurrency
> > > level, 50
> > > > > > messages per thread, I'm getting the following exception several
> > > times
> > > > > from
> > > > > > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> > > > > output.
> > > > > > No exceptions are displayed from the server side. Hence, I think
the
> > > > > server
> > > > > > is rejecting some socket connections because it's already fully
> > > loaded.
> > > > > >
> > > > > > I'm running the server and load generator on two server machines
> > > that are
> > > > > > connected with Gigabit Ethernet. Each server has 16 CPU cores,
16 GB
> > > > > memory
> > > > > > etc.
> > > > > >
> > > > > > I have increased the socket timeout in both httpcore-ab and
server
> > > side
> > > > > > ListeningIOReactor to 360000. 16 threads are allocated for the
> > > reactor.
> > > > > > Messages received by the server are processed in a separate
worker
> > > pool
> > > > > > which has ~500 threads. I have also increased the open file
limit of
> > > my
> > > > > OS
> > > > > > (Debian) to 65535.
> > > > > >
> > > > > > Is there any tuning parameter of HTTPCore that I can used to
get rid
> > > of
> > > > > > this exception? Perhaps, something similar to "acceptQueueSize"
in
> > > Jetty
> > > > > >  [2]?
> > > > > >
> > > > >
> > > > > Hi Sadeep
> > > > >
> > > > > A couple of points in no particular logical order.
> > > > >
> > > > > (1) HttpCore does not depend on a logging toolkit and does not do
any
> > > > > logging per default. If it ever encounters an abnormal situation,
> > > > > instead of logging such an event it throws an exception and terminates.
> > > > > Quite often this is not desirable, so one can provide an exception
> > > > > handler that can log using a a logging toolkit of choice or handle
a
> > > > > particular type of exception in an application specific way.
> > > > >
> > > > > If the server did not terminate under with an exception, I kind of
> > > > > suspect it was still trying to chew through the load without
> > > > > encountering anything abnormal. Unless you have added a custom
> > > exception
> > > > > handler.
> > > > >
> > > > > (2) HttpCore does not have a concept of accepted connection queue.
As
> > > > > soon as the listener accepts an incoming connection its channel gets
> > > > > immediately added to the underlying i/o selector. No intermediate
> > > > > queuing takes place.
> > > >
> > > >
> > > > > (3) Given that the connection times out on the client, this sounds
more
> > > > > like a client side issue. Was connection really suck in
> > > > > Socket#socketConnect for 360000?
> > > > >
> > > >
> > > > I'm using httpcore-ab [1] as the client. I have set socket timeout of
> > > using
> > > > -t 360000, and changed the connection timeout with the patch shown below.
> > > > But I'm starting to get this error well before 360000 millis past the
> > > test
> > > > startup. Therefore, I there is no possibility that the connection attempt
> > > > hangs for 360000 millis before printing this exception. I was thinking
> > > > whether there are upper bounds for these limits that cause my settings
to
> > > > be neglected.
> > > >
> > >
> > > You see. If java.net.ConnectException: Connection timed out is thrown by
> > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> > > there must be something wrong with the client side.
> > >
> > 
> > Could this be a possible bug in httpcore-ab code? I'm quite sure that it
> > doesn't wait 360000 millis before throwing this exception.
> > 
> 

Hi Sadeep

It did turn out to be an issue with the way HttpCore AB initialized
connections. It was basically using an old (Java 1.0) socket constructor
whose behavior can differ between OS platforms and possibly between JRE
major releases with regards to connect timeout handling. The connect
timeout value set in the HttpParams simply had no effect and therefore
connections in your tests were timing out before 360000 milliseconds
elapsed (probably using some sort of a system default). 

I committed a fix for the problem

http://svn.apache.org/viewvc?rev=1238566&view=rev

Please review

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message