hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sadeep Jayasumana <gayansad...@gmail.com>
Subject Re: HTTPCore-NIO: Server rejecting socket connections?
Date Tue, 07 Feb 2012 13:24:01 GMT
Hi Oleg,

On Wed, Feb 1, 2012 at 5:38 PM, Oleg Kalnichevski <olegk@apache.org> wrote:

> On Wed, 2012-02-01 at 16:16 +0530, Sadeep Jayasumana wrote:
> > Hi Oleg,
> >
> > I tested the patch. When I haven't explicitly set the connection
> timeout, I
> > got the same exception, java.net.ConnectException: Connection timed out,
> at
> > concurrency = 3000, messages/thread = 100 intermittently. One time I got
> it
> > in less than 17 seconds. This is contrary to the Javadoc of
> > Socket#connect() method [1] which says "*A timeout of zero is interpreted
> > as an infinite timeout*".
> >
> > Then when I explicitly set it to 5 seconds as follows I was unable to
> > reproduce the exception although I tried several times.
> >
> > socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5);
> >
> > The other doubt I have is; although I'm getting a
> > java.net.ConnectException, according to [1], this method should throw
> > SocketTimeoutException in case of a timeout. Further, JavaDoc of
> > ConnectException (the one I'm actually getting) says it signals that *"an
> > error occurred while attempting to connect a socket to a remote address
> and
> > port. Typically, the connection was refused remotely (e.g., no process is
> > listening on the remote address/port)." *
> >
> > Therefore, I believe that the actual cause of this error is server's
> > ListeningIOReactor not being able to react to incoming connection
> requests.
> > I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor.
> > Maybe increasing it further would solve this issue. I will test this
> > further and let you know.
> >
>
> ConnectException is a subclass of SocketTimeoutException and the queer
> thing the exception message clearly says "Connection timed out".


It turns out that ConnectException is not a subclass of
SocketTimeoutException. That's why I'm confused.

So, I
> am still not entirely convinced this rules out a client side issue.
>
> Please also note that in its current implementation the
> DefaultListeningIOReactor utilizes only one thread and only one i/o
> selector for all incoming requests no matter how many worker threads are
> allocated for handling i/o on active channels. If you suspect this could
> be the issue, you may need to consider writing a custom
> ListeningIOReactor
>

Got the idea. Thanks for the explanation.

Thanks,
Sadeep


>
> Oleg
>
> > [1]
> >
> http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29
> > [2]
> http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html
> >
> > Thanks,
> > Sadeep
> >
> > On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana <
> gayansadeep@gmail.com>wrote:
> >
> > > Hi Oleg,
> > >
> > > On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <olegk@apache.org
> >wrote:
> > >
> > >> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
> > >> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> > >> > > Hi,
> > >> > >
> > >> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <
> olegk@apache.org>
> > >> wrote:
> > >> > >
> > >> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski
<
> > >> olegk@apache.org>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana
wrote:
> > >> > > > > > > Hi,
> > >> > > > > > >
> > >> > > > > > > I have written an HTTP server using HTTPCore-NIO
and
> currently
> > >> > > > testing it
> > >> > > > > > > with httpcore-ab [1]. I have been seeing
very good TPS
> values
> > >> so
> > >> > > > far. But
> > >> > > > > > > when I try to send 5KB HTTP messages with
2500 and
> concurrency
> > >> > > > level, 50
> > >> > > > > > > messages per thread, I'm getting the following
exception
> > >> several
> > >> > > > times
> > >> > > > > > from
> > >> > > > > > > httpcore-ab and it results in "Failed requests"
in the
> > >> httpcore-ab
> > >> > > > > > output.
> > >> > > > > > > No exceptions are displayed from the server
side. Hence, I
> > >> think the
> > >> > > > > > server
> > >> > > > > > > is rejecting some socket connections because
it's already
> > >> fully
> > >> > > > loaded.
> > >> > > > > > >
> > >> > > > > > > I'm running the server and load generator
on two server
> > >> machines
> > >> > > > that are
> > >> > > > > > > connected with Gigabit Ethernet. Each server
has 16 CPU
> > >> cores, 16 GB
> > >> > > > > > memory
> > >> > > > > > > etc.
> > >> > > > > > >
> > >> > > > > > > I have increased the socket timeout in both
httpcore-ab
> and
> > >> server
> > >> > > > side
> > >> > > > > > > ListeningIOReactor to 360000. 16 threads
are allocated
> for the
> > >> > > > reactor.
> > >> > > > > > > Messages received by the server are processed
in a
> separate
> > >> worker
> > >> > > > pool
> > >> > > > > > > which has ~500 threads. I have also increased
the open
> file
> > >> limit of
> > >> > > > my
> > >> > > > > > OS
> > >> > > > > > > (Debian) to 65535.
> > >> > > > > > >
> > >> > > > > > > Is there any tuning parameter of HTTPCore
that I can used
> to
> > >> get rid
> > >> > > > of
> > >> > > > > > > this exception? Perhaps, something similar
to
> > >> "acceptQueueSize" in
> > >> > > > Jetty
> > >> > > > > > >  [2]?
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > Hi Sadeep
> > >> > > > > >
> > >> > > > > > A couple of points in no particular logical order.
> > >> > > > > >
> > >> > > > > > (1) HttpCore does not depend on a logging toolkit
and does
> not
> > >> do any
> > >> > > > > > logging per default. If it ever encounters an
abnormal
> > >> situation,
> > >> > > > > > instead of logging such an event it throws an
exception and
> > >> terminates.
> > >> > > > > > Quite often this is not desirable, so one can
provide an
> > >> exception
> > >> > > > > > handler that can log using a a logging toolkit
of choice or
> > >> handle a
> > >> > > > > > particular type of exception in an application
specific way.
> > >> > > > > >
> > >> > > > > > If the server did not terminate under with an
exception, I
> kind
> > >> of
> > >> > > > > > suspect it was still trying to chew through the
load without
> > >> > > > > > encountering anything abnormal. Unless you have
added a
> custom
> > >> > > > exception
> > >> > > > > > handler.
> > >> > > > > >
> > >> > > > > > (2) HttpCore does not have a concept of accepted
connection
> > >> queue. As
> > >> > > > > > soon as the listener accepts an incoming connection
its
> channel
> > >> gets
> > >> > > > > > immediately added to the underlying i/o selector.
No
> > >> intermediate
> > >> > > > > > queuing takes place.
> > >> > > > >
> > >> > > > >
> > >> > > > > > (3) Given that the connection times out on the
client, this
> > >> sounds more
> > >> > > > > > like a client side issue. Was connection really
suck in
> > >> > > > > > Socket#socketConnect for 360000?
> > >> > > > > >
> > >> > > > >
> > >> > > > > I'm using httpcore-ab [1] as the client. I have set
socket
> > >> timeout of
> > >> > > > using
> > >> > > > > -t 360000, and changed the connection timeout with
the patch
> > >> shown below.
> > >> > > > > But I'm starting to get this error well before 360000
millis
> past
> > >> the
> > >> > > > test
> > >> > > > > startup. Therefore, I there is no possibility that
the
> connection
> > >> attempt
> > >> > > > > hangs for 360000 millis before printing this exception.
I was
> > >> thinking
> > >> > > > > whether there are upper bounds for these limits that
cause my
> > >> settings to
> > >> > > > > be neglected.
> > >> > > > >
> > >> > > >
> > >> > > > You see. If java.net.ConnectException: Connection timed
out is
> > >> thrown by
> > >> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis
> elapse,
> > >> > > > there must be something wrong with the client side.
> > >> > > >
> > >> > >
> > >> > > Could this be a possible bug in httpcore-ab code? I'm quite sure
> that
> > >> it
> > >> > > doesn't wait 360000 millis before throwing this exception.
> > >> > >
> > >> >
> > >>
> > >> Hi Sadeep
> > >>
> > >> It did turn out to be an issue with the way HttpCore AB initialized
> > >> connections. It was basically using an old (Java 1.0) socket
> constructor
> > >> whose behavior can differ between OS platforms and possibly between
> JRE
> > >> major releases with regards to connect timeout handling. The connect
> > >> timeout value set in the HttpParams simply had no effect and therefore
> > >> connections in your tests were timing out before 360000 milliseconds
> > >> elapsed (probably using some sort of a system default).
> > >>
> > >> I committed a fix for the problem
> > >>
> > >> http://svn.apache.org/viewvc?rev=1238566&view=rev
> > >>
> > >> Please review
> > >>
> > >
> > > Many thanks for looking into this. I will review this at my earliest
> and
> > > let you know the outcome.
> > >
> > > Thanks,
> > > Sadeep
> > >
> > >
> > >>
> > >> Oleg
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > >> For additional commands, e-mail: dev-help@hc.apache.org
> > >>
> > >>
> > >
> > >
> > > --
> > >
> > > Sadeep Jayasumana****
> > >
> > > Email: gayansadeep@gmail.com****
> > >
> > > Phone: +94-77-2266507
> > >
> > >
> > >
> > >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>


-- 

Sadeep Jayasumana****

Email: gayansadeep@gmail.com****

Phone: +94-77-2266507

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message