hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: HTTPCore-NIO: Server rejecting socket connections?
Date Wed, 01 Feb 2012 12:08:22 GMT
On Wed, 2012-02-01 at 16:16 +0530, Sadeep Jayasumana wrote:
> Hi Oleg,
> 
> I tested the patch. When I haven't explicitly set the connection timeout, I
> got the same exception, java.net.ConnectException: Connection timed out, at
> concurrency = 3000, messages/thread = 100 intermittently. One time I got it
> in less than 17 seconds. This is contrary to the Javadoc of
> Socket#connect() method [1] which says "*A timeout of zero is interpreted
> as an infinite timeout*".
> 
> Then when I explicitly set it to 5 seconds as follows I was unable to
> reproduce the exception although I tried several times.
> 
> socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5);
> 
> The other doubt I have is; although I'm getting a
> java.net.ConnectException, according to [1], this method should throw
> SocketTimeoutException in case of a timeout. Further, JavaDoc of
> ConnectException (the one I'm actually getting) says it signals that *"an
> error occurred while attempting to connect a socket to a remote address and
> port. Typically, the connection was refused remotely (e.g., no process is
> listening on the remote address/port)." *
> 
> Therefore, I believe that the actual cause of this error is server's
> ListeningIOReactor not being able to react to incoming connection requests.
> I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor.
> Maybe increasing it further would solve this issue. I will test this
> further and let you know.
> 

ConnectException is a subclass of SocketTimeoutException and the queer
thing the exception message clearly says "Connection timed out". So, I
am still not entirely convinced this rules out a client side issue. 

Please also note that in its current implementation the
DefaultListeningIOReactor utilizes only one thread and only one i/o
selector for all incoming requests no matter how many worker threads are
allocated for handling i/o on active channels. If you suspect this could
be the issue, you may need to consider writing a custom
ListeningIOReactor

Oleg

> [1]
> http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29
> [2] http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html
> 
> Thanks,
> Sadeep
> 
> On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana <gayansadeep@gmail.com>wrote:
> 
> > Hi Oleg,
> >
> > On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <olegk@apache.org>wrote:
> >
> >> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
> >> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> >> > > Hi,
> >> > >
> >> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <olegk@apache.org>
> >> wrote:
> >> > >
> >> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> >> > > > > Hi,
> >> > > > >
> >> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <
> >> olegk@apache.org>
> >> > > > wrote:
> >> > > > >
> >> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana
wrote:
> >> > > > > > > Hi,
> >> > > > > > >
> >> > > > > > > I have written an HTTP server using HTTPCore-NIO
and currently
> >> > > > testing it
> >> > > > > > > with httpcore-ab [1]. I have been seeing very
good TPS values
> >> so
> >> > > > far. But
> >> > > > > > > when I try to send 5KB HTTP messages with 2500
and concurrency
> >> > > > level, 50
> >> > > > > > > messages per thread, I'm getting the following
exception
> >> several
> >> > > > times
> >> > > > > > from
> >> > > > > > > httpcore-ab and it results in "Failed requests"
in the
> >> httpcore-ab
> >> > > > > > output.
> >> > > > > > > No exceptions are displayed from the server side.
Hence, I
> >> think the
> >> > > > > > server
> >> > > > > > > is rejecting some socket connections because it's
already
> >> fully
> >> > > > loaded.
> >> > > > > > >
> >> > > > > > > I'm running the server and load generator on two
server
> >> machines
> >> > > > that are
> >> > > > > > > connected with Gigabit Ethernet. Each server has
16 CPU
> >> cores, 16 GB
> >> > > > > > memory
> >> > > > > > > etc.
> >> > > > > > >
> >> > > > > > > I have increased the socket timeout in both httpcore-ab
and
> >> server
> >> > > > side
> >> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated
for the
> >> > > > reactor.
> >> > > > > > > Messages received by the server are processed
in a separate
> >> worker
> >> > > > pool
> >> > > > > > > which has ~500 threads. I have also increased
the open file
> >> limit of
> >> > > > my
> >> > > > > > OS
> >> > > > > > > (Debian) to 65535.
> >> > > > > > >
> >> > > > > > > Is there any tuning parameter of HTTPCore that
I can used to
> >> get rid
> >> > > > of
> >> > > > > > > this exception? Perhaps, something similar to
> >> "acceptQueueSize" in
> >> > > > Jetty
> >> > > > > > >  [2]?
> >> > > > > > >
> >> > > > > >
> >> > > > > > Hi Sadeep
> >> > > > > >
> >> > > > > > A couple of points in no particular logical order.
> >> > > > > >
> >> > > > > > (1) HttpCore does not depend on a logging toolkit and
does not
> >> do any
> >> > > > > > logging per default. If it ever encounters an abnormal
> >> situation,
> >> > > > > > instead of logging such an event it throws an exception
and
> >> terminates.
> >> > > > > > Quite often this is not desirable, so one can provide
an
> >> exception
> >> > > > > > handler that can log using a a logging toolkit of choice
or
> >> handle a
> >> > > > > > particular type of exception in an application specific
way.
> >> > > > > >
> >> > > > > > If the server did not terminate under with an exception,
I kind
> >> of
> >> > > > > > suspect it was still trying to chew through the load
without
> >> > > > > > encountering anything abnormal. Unless you have added
a custom
> >> > > > exception
> >> > > > > > handler.
> >> > > > > >
> >> > > > > > (2) HttpCore does not have a concept of accepted connection
> >> queue. As
> >> > > > > > soon as the listener accepts an incoming connection
its channel
> >> gets
> >> > > > > > immediately added to the underlying i/o selector. No
> >> intermediate
> >> > > > > > queuing takes place.
> >> > > > >
> >> > > > >
> >> > > > > > (3) Given that the connection times out on the client,
this
> >> sounds more
> >> > > > > > like a client side issue. Was connection really suck
in
> >> > > > > > Socket#socketConnect for 360000?
> >> > > > > >
> >> > > > >
> >> > > > > I'm using httpcore-ab [1] as the client. I have set socket
> >> timeout of
> >> > > > using
> >> > > > > -t 360000, and changed the connection timeout with the patch
> >> shown below.
> >> > > > > But I'm starting to get this error well before 360000 millis
past
> >> the
> >> > > > test
> >> > > > > startup. Therefore, I there is no possibility that the connection
> >> attempt
> >> > > > > hangs for 360000 millis before printing this exception.
I was
> >> thinking
> >> > > > > whether there are upper bounds for these limits that cause
my
> >> settings to
> >> > > > > be neglected.
> >> > > > >
> >> > > >
> >> > > > You see. If java.net.ConnectException: Connection timed out is
> >> thrown by
> >> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> >> > > > there must be something wrong with the client side.
> >> > > >
> >> > >
> >> > > Could this be a possible bug in httpcore-ab code? I'm quite sure that
> >> it
> >> > > doesn't wait 360000 millis before throwing this exception.
> >> > >
> >> >
> >>
> >> Hi Sadeep
> >>
> >> It did turn out to be an issue with the way HttpCore AB initialized
> >> connections. It was basically using an old (Java 1.0) socket constructor
> >> whose behavior can differ between OS platforms and possibly between JRE
> >> major releases with regards to connect timeout handling. The connect
> >> timeout value set in the HttpParams simply had no effect and therefore
> >> connections in your tests were timing out before 360000 milliseconds
> >> elapsed (probably using some sort of a system default).
> >>
> >> I committed a fix for the problem
> >>
> >> http://svn.apache.org/viewvc?rev=1238566&view=rev
> >>
> >> Please review
> >>
> >
> > Many thanks for looking into this. I will review this at my earliest and
> > let you know the outcome.
> >
> > Thanks,
> > Sadeep
> >
> >
> >>
> >> Oleg
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> >> For additional commands, e-mail: dev-help@hc.apache.org
> >>
> >>
> >
> >
> > --
> >
> > Sadeep Jayasumana****
> >
> > Email: gayansadeep@gmail.com****
> >
> > Phone: +94-77-2266507
> >
> >
> >
> >
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message