hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: HTTPCore-NIO: Server rejecting socket connections?
Date Mon, 30 Jan 2012 13:56:37 GMT
On Mon, 2012-01-30 at 17:45 +0530, Sadeep Jayasumana wrote:
> Hi,
> 
> On Mon, Jan 30, 2012 at 5:15 PM, Oleg Kalnichevski <olegk@apache.org> wrote:
> 
> > On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> > > Hi,
> > >
> > > On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <olegk@apache.org>
> > wrote:
> > >
> > > > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > > > Hi,
> > > > >
> > > > > I have written an HTTP server using HTTPCore-NIO and currently
> > testing it
> > > > > with httpcore-ab [1]. I have been seeing very good TPS values so
> > far. But
> > > > > when I try to send 5KB HTTP messages with 2500 and concurrency
> > level, 50
> > > > > messages per thread, I'm getting the following exception several
> > times
> > > > from
> > > > > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> > > > output.
> > > > > No exceptions are displayed from the server side. Hence, I think
the
> > > > server
> > > > > is rejecting some socket connections because it's already fully
> > loaded.
> > > > >
> > > > > I'm running the server and load generator on two server machines
> > that are
> > > > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16
GB
> > > > memory
> > > > > etc.
> > > > >
> > > > > I have increased the socket timeout in both httpcore-ab and server
> > side
> > > > > ListeningIOReactor to 360000. 16 threads are allocated for the
> > reactor.
> > > > > Messages received by the server are processed in a separate worker
> > pool
> > > > > which has ~500 threads. I have also increased the open file limit
of
> > my
> > > > OS
> > > > > (Debian) to 65535.
> > > > >
> > > > > Is there any tuning parameter of HTTPCore that I can used to get
rid
> > of
> > > > > this exception? Perhaps, something similar to "acceptQueueSize" in
> > Jetty
> > > > >  [2]?
> > > > >
> > > >
> > > > Hi Sadeep
> > > >
> > > > A couple of points in no particular logical order.
> > > >
> > > > (1) HttpCore does not depend on a logging toolkit and does not do any
> > > > logging per default. If it ever encounters an abnormal situation,
> > > > instead of logging such an event it throws an exception and terminates.
> > > > Quite often this is not desirable, so one can provide an exception
> > > > handler that can log using a a logging toolkit of choice or handle a
> > > > particular type of exception in an application specific way.
> > > >
> > > > If the server did not terminate under with an exception, I kind of
> > > > suspect it was still trying to chew through the load without
> > > > encountering anything abnormal. Unless you have added a custom
> > exception
> > > > handler.
> > > >
> > > > (2) HttpCore does not have a concept of accepted connection queue. As
> > > > soon as the listener accepts an incoming connection its channel gets
> > > > immediately added to the underlying i/o selector. No intermediate
> > > > queuing takes place.
> > >
> > >
> > > > (3) Given that the connection times out on the client, this sounds more
> > > > like a client side issue. Was connection really suck in
> > > > Socket#socketConnect for 360000?
> > > >
> > >
> > > I'm using httpcore-ab [1] as the client. I have set socket timeout of
> > using
> > > -t 360000, and changed the connection timeout with the patch shown below.
> > > But I'm starting to get this error well before 360000 millis past the
> > test
> > > startup. Therefore, I there is no possibility that the connection attempt
> > > hangs for 360000 millis before printing this exception. I was thinking
> > > whether there are upper bounds for these limits that cause my settings to
> > > be neglected.
> > >
> >
> > You see. If java.net.ConnectException: Connection timed out is thrown by
> > java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
> > there must be something wrong with the client side.
> >
> 
> Could this be a possible bug in httpcore-ab code? I'm quite sure that it
> doesn't wait 360000 millis before throwing this exception.
> 

Of course, it is not impossible. However, if you take look at the
exception stack trace, the exception originates from native part of JRE
code, which is responsible for low level TCP/IP functionality. 

Have you tried using an infinite timeout (no timeout)?

Another thing can think of is trying a different client side JRE. 

Oleg

> Thanks,
> Sadeep
> 
> 
> >
> > Here's the contract of the method:
> > --------------------------------
> > Connects this socket to the server with a specified timeout value. A
> > timeout of zero is interpreted as an infinite timeout. The connection
> > will then block until established or an error occurs.
> >
> >    Parameters:
> >        endpoint - the SocketAddress
> >        timeout - the timeout value to be used in milliseconds.
> >    Throws:
> >        IOException - if an error occurs during the connection
> >        SocketTimeoutException - if timeout expires before connecting
> >        IllegalBlockingModeException - if this socket has an associated
> > channel, and the channel is in non-blocking mode
> >        IllegalArgumentException - if endpoint is null or is a
> > SocketAddress subclass not supported by this socket
> > --------------------------------
> >
> > ConnectException is a subclass of SocketTimeoutException. My
> > understanding of the contract is that the exception cannot be thrown
> > unless the timeout in milliseconds fully elapses.
> >
> > Oleg
> >
> > > --- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision
> > > 1205292)
> > > +++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working
> > > copy)
> > > @@ -227,7 +227,8 @@
> > >              .setParameter(HttpProtocolParams.USER_AGENT,
> > "HttpCore-AB/1.1")
> > >              .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE,
> > > useExpectContinue)
> > >
> > >  .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false)
> > > -            .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> > > socketTimeout);
> > > +            .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> > > socketTimeout)
> > > +            .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT,
> > 360
> > > * 1000);
> > >          return params;
> > >      }
> > >  }
> > >
> > >
> > > >
> > > > (4) Quite often it is useful to see what it really going on on the I/O
> > > > reactor level. You might want to add I/O event and wire logging by
> > > > copying logging components from the contrib package [1]
> > > >
> > >
> > > I could do this. But will wire logging really help at this concurrency
> > > level and messages/thread?
> > >
> > > [1]
> > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > >
> > > Thanks,
> > > Sadeep
> > >
> > > >
> > > > I hope this helps somewhat. In any way I can say I am quite sure
> > > > HttpCore does not reject incoming connections no matter what.
> > >
> > >
> > > > Oleg
> > > >
> > > > [1]
> > > >
> > > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
> > > >
> > > > > [1]
> > > > >
> > > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > > > > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> > > > > [3]
> > > > > java.net.ConnectException: Connection timed out
> > > > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > > > at
> > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > > > at java.net.Socket.connect(Socket.java:529)
> > > > > at java.net.Socket.connect(Socket.java:478)
> > > > > at java.net.Socket.<init>(Socket.java:375)
> > > > > at java.net.Socket.<init>(Socket.java:189)
> > > > > at
> > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > java.net.ConnectException: Connection timed out
> > > > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > > > at
> > java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > > > at java.net.Socket.connect(Socket.java:529)
> > > > > at java.net.Socket.connect(Socket.java:478)
> > > > > at java.net.Socket.<init>(Socket.java:375)
> > > > > at java.net.Socket.<init>(Socket.java:189)
> > > > > at
> > > > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > at
> > > > >
> > > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > at java.lang.Thread.run(Thread.java:662)
> > > > >
> > > > > Thanks & Regards,
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > > >
> > >
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
> >
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message