hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sadeep Jayasumana <gayansad...@gmail.com>
Subject Re: HTTPCore-NIO: Server rejecting socket connections?
Date Wed, 01 Feb 2012 10:46:42 GMT
Hi Oleg,

I tested the patch. When I haven't explicitly set the connection timeout, I
got the same exception, java.net.ConnectException: Connection timed out, at
concurrency = 3000, messages/thread = 100 intermittently. One time I got it
in less than 17 seconds. This is contrary to the Javadoc of
Socket#connect() method [1] which says "*A timeout of zero is interpreted
as an infinite timeout*".

Then when I explicitly set it to 5 seconds as follows I was unable to
reproduce the exception although I tried several times.

socket.connect(new InetSocketAddress(hostname, port), 1000 * 60 * 5);

The other doubt I have is; although I'm getting a
java.net.ConnectException, according to [1], this method should throw
SocketTimeoutException in case of a timeout. Further, JavaDoc of
ConnectException (the one I'm actually getting) says it signals that *"an
error occurred while attempting to connect a socket to a remote address and
port. Typically, the connection was refused remotely (e.g., no process is
listening on the remote address/port)." *

Therefore, I believe that the actual cause of this error is server's
ListeningIOReactor not being able to react to incoming connection requests.
I had allocated 16 threads (= no of CPU cores) for ListeningIOReactor.
Maybe increasing it further would solve this issue. I will test this
further and let you know.

[1]
http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#connect%28java.net.SocketAddress,%20int%29
[2] http://docs.oracle.com/javase/6/docs/api/java/net/ConnectException.html

Thanks,
Sadeep

On Tue, Jan 31, 2012 at 9:42 PM, Sadeep Jayasumana <gayansadeep@gmail.com>wrote:

> Hi Oleg,
>
> On Tue, Jan 31, 2012 at 7:48 PM, Oleg Kalnichevski <olegk@apache.org>wrote:
>
>> On Mon, 2012-01-30 at 14:56 +0100, Oleg Kalnichevski wrote:
>> > On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
>> > > Hi,
>> > >
>> > > On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <olegk@apache.org>
>> wrote:
>> > >
>> > > > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
>> > > > > Hi,
>> > > > >
>> > > > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <
>> olegk@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > I have written an HTTP server using HTTPCore-NIO and
currently
>> > > > testing it
>> > > > > > > with httpcore-ab [1]. I have been seeing very good
TPS values
>> so
>> > > > far. But
>> > > > > > > when I try to send 5KB HTTP messages with 2500 and
concurrency
>> > > > level, 50
>> > > > > > > messages per thread, I'm getting the following exception
>> several
>> > > > times
>> > > > > > from
>> > > > > > > httpcore-ab and it results in "Failed requests" in
the
>> httpcore-ab
>> > > > > > output.
>> > > > > > > No exceptions are displayed from the server side. Hence,
I
>> think the
>> > > > > > server
>> > > > > > > is rejecting some socket connections because it's already
>> fully
>> > > > loaded.
>> > > > > > >
>> > > > > > > I'm running the server and load generator on two server
>> machines
>> > > > that are
>> > > > > > > connected with Gigabit Ethernet. Each server has 16
CPU
>> cores, 16 GB
>> > > > > > memory
>> > > > > > > etc.
>> > > > > > >
>> > > > > > > I have increased the socket timeout in both httpcore-ab
and
>> server
>> > > > side
>> > > > > > > ListeningIOReactor to 360000. 16 threads are allocated
for the
>> > > > reactor.
>> > > > > > > Messages received by the server are processed in a
separate
>> worker
>> > > > pool
>> > > > > > > which has ~500 threads. I have also increased the open
file
>> limit of
>> > > > my
>> > > > > > OS
>> > > > > > > (Debian) to 65535.
>> > > > > > >
>> > > > > > > Is there any tuning parameter of HTTPCore that I can
used to
>> get rid
>> > > > of
>> > > > > > > this exception? Perhaps, something similar to
>> "acceptQueueSize" in
>> > > > Jetty
>> > > > > > >  [2]?
>> > > > > > >
>> > > > > >
>> > > > > > Hi Sadeep
>> > > > > >
>> > > > > > A couple of points in no particular logical order.
>> > > > > >
>> > > > > > (1) HttpCore does not depend on a logging toolkit and does
not
>> do any
>> > > > > > logging per default. If it ever encounters an abnormal
>> situation,
>> > > > > > instead of logging such an event it throws an exception
and
>> terminates.
>> > > > > > Quite often this is not desirable, so one can provide an
>> exception
>> > > > > > handler that can log using a a logging toolkit of choice
or
>> handle a
>> > > > > > particular type of exception in an application specific
way.
>> > > > > >
>> > > > > > If the server did not terminate under with an exception,
I kind
>> of
>> > > > > > suspect it was still trying to chew through the load without
>> > > > > > encountering anything abnormal. Unless you have added a
custom
>> > > > exception
>> > > > > > handler.
>> > > > > >
>> > > > > > (2) HttpCore does not have a concept of accepted connection
>> queue. As
>> > > > > > soon as the listener accepts an incoming connection its
channel
>> gets
>> > > > > > immediately added to the underlying i/o selector. No
>> intermediate
>> > > > > > queuing takes place.
>> > > > >
>> > > > >
>> > > > > > (3) Given that the connection times out on the client, this
>> sounds more
>> > > > > > like a client side issue. Was connection really suck in
>> > > > > > Socket#socketConnect for 360000?
>> > > > > >
>> > > > >
>> > > > > I'm using httpcore-ab [1] as the client. I have set socket
>> timeout of
>> > > > using
>> > > > > -t 360000, and changed the connection timeout with the patch
>> shown below.
>> > > > > But I'm starting to get this error well before 360000 millis
past
>> the
>> > > > test
>> > > > > startup. Therefore, I there is no possibility that the connection
>> attempt
>> > > > > hangs for 360000 millis before printing this exception. I was
>> thinking
>> > > > > whether there are upper bounds for these limits that cause my
>> settings to
>> > > > > be neglected.
>> > > > >
>> > > >
>> > > > You see. If java.net.ConnectException: Connection timed out is
>> thrown by
>> > > > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
>> > > > there must be something wrong with the client side.
>> > > >
>> > >
>> > > Could this be a possible bug in httpcore-ab code? I'm quite sure that
>> it
>> > > doesn't wait 360000 millis before throwing this exception.
>> > >
>> >
>>
>> Hi Sadeep
>>
>> It did turn out to be an issue with the way HttpCore AB initialized
>> connections. It was basically using an old (Java 1.0) socket constructor
>> whose behavior can differ between OS platforms and possibly between JRE
>> major releases with regards to connect timeout handling. The connect
>> timeout value set in the HttpParams simply had no effect and therefore
>> connections in your tests were timing out before 360000 milliseconds
>> elapsed (probably using some sort of a system default).
>>
>> I committed a fix for the problem
>>
>> http://svn.apache.org/viewvc?rev=1238566&view=rev
>>
>> Please review
>>
>
> Many thanks for looking into this. I will review this at my earliest and
> let you know the outcome.
>
> Thanks,
> Sadeep
>
>
>>
>> Oleg
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>>
>
>
> --
>
> Sadeep Jayasumana****
>
> Email: gayansadeep@gmail.com****
>
> Phone: +94-77-2266507
>
>
>
>


-- 

Sadeep Jayasumana****

Email: gayansadeep@gmail.com****

Phone: +94-77-2266507

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message