hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: HTTPCore-NIO: Server rejecting socket connections?
Date Mon, 30 Jan 2012 11:45:48 GMT
On Sat, 2012-01-28 at 23:46 +0530, Sadeep Jayasumana wrote:
> Hi,
> 
> On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <olegk@apache.org> wrote:
> 
> > On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > > Hi,
> > >
> > > I have written an HTTP server using HTTPCore-NIO and currently testing it
> > > with httpcore-ab [1]. I have been seeing very good TPS values so far. But
> > > when I try to send 5KB HTTP messages with 2500 and concurrency level, 50
> > > messages per thread, I'm getting the following exception several times
> > from
> > > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> > output.
> > > No exceptions are displayed from the server side. Hence, I think the
> > server
> > > is rejecting some socket connections because it's already fully loaded.
> > >
> > > I'm running the server and load generator on two server machines that are
> > > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB
> > memory
> > > etc.
> > >
> > > I have increased the socket timeout in both httpcore-ab and server side
> > > ListeningIOReactor to 360000. 16 threads are allocated for the reactor.
> > > Messages received by the server are processed in a separate worker pool
> > > which has ~500 threads. I have also increased the open file limit of my
> > OS
> > > (Debian) to 65535.
> > >
> > > Is there any tuning parameter of HTTPCore that I can used to get rid of
> > > this exception? Perhaps, something similar to "acceptQueueSize" in Jetty
> > >  [2]?
> > >
> >
> > Hi Sadeep
> >
> > A couple of points in no particular logical order.
> >
> > (1) HttpCore does not depend on a logging toolkit and does not do any
> > logging per default. If it ever encounters an abnormal situation,
> > instead of logging such an event it throws an exception and terminates.
> > Quite often this is not desirable, so one can provide an exception
> > handler that can log using a a logging toolkit of choice or handle a
> > particular type of exception in an application specific way.
> >
> > If the server did not terminate under with an exception, I kind of
> > suspect it was still trying to chew through the load without
> > encountering anything abnormal. Unless you have added a custom exception
> > handler.
> >
> > (2) HttpCore does not have a concept of accepted connection queue. As
> > soon as the listener accepts an incoming connection its channel gets
> > immediately added to the underlying i/o selector. No intermediate
> > queuing takes place.
> 
> 
> > (3) Given that the connection times out on the client, this sounds more
> > like a client side issue. Was connection really suck in
> > Socket#socketConnect for 360000?
> >
> 
> I'm using httpcore-ab [1] as the client. I have set socket timeout of using
> -t 360000, and changed the connection timeout with the patch shown below.
> But I'm starting to get this error well before 360000 millis past the test
> startup. Therefore, I there is no possibility that the connection attempt
> hangs for 360000 millis before printing this exception. I was thinking
> whether there are upper bounds for these limits that cause my settings to
> be neglected.
> 

You see. If java.net.ConnectException: Connection timed out is thrown by
java.net.PlainSocketImpl.socketConnect before 360000 millis elapse,
there must be something wrong with the client side. 

Here's the contract of the method:
--------------------------------
Connects this socket to the server with a specified timeout value. A
timeout of zero is interpreted as an infinite timeout. The connection
will then block until established or an error occurs.

    Parameters:
        endpoint - the SocketAddress
        timeout - the timeout value to be used in milliseconds. 
    Throws:
        IOException - if an error occurs during the connection 
        SocketTimeoutException - if timeout expires before connecting 
        IllegalBlockingModeException - if this socket has an associated
channel, and the channel is in non-blocking mode 
        IllegalArgumentException - if endpoint is null or is a
SocketAddress subclass not supported by this socket
--------------------------------

ConnectException is a subclass of SocketTimeoutException. My
understanding of the contract is that the exception cannot be thrown
unless the timeout in milliseconds fully elapses.

Oleg

> --- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision
> 1205292)
> +++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working
> copy)
> @@ -227,7 +227,8 @@
>              .setParameter(HttpProtocolParams.USER_AGENT, "HttpCore-AB/1.1")
>              .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE,
> useExpectContinue)
> 
>  .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false)
> -            .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> socketTimeout);
> +            .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
> socketTimeout)
> +            .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT, 360
> * 1000);
>          return params;
>      }
>  }
> 
> 
> >
> > (4) Quite often it is useful to see what it really going on on the I/O
> > reactor level. You might want to add I/O event and wire logging by
> > copying logging components from the contrib package [1]
> >
> 
> I could do this. But will wire logging really help at this concurrency
> level and messages/thread?
> 
> [1]
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> 
> Thanks,
> Sadeep
> 
> >
> > I hope this helps somewhat. In any way I can say I am quite sure
> > HttpCore does not reject incoming connections no matter what.
> 
> 
> > Oleg
> >
> > [1]
> >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
> >
> > > [1]
> > >
> > https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> > > [3]
> > > java.net.ConnectException: Connection timed out
> > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > at java.net.Socket.connect(Socket.java:529)
> > > at java.net.Socket.connect(Socket.java:478)
> > > at java.net.Socket.<init>(Socket.java:375)
> > > at java.net.Socket.<init>(Socket.java:189)
> > > at
> > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > java.net.ConnectException: Connection timed out
> > > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > > at java.net.Socket.connect(Socket.java:529)
> > > at java.net.Socket.connect(Socket.java:478)
> > > at java.net.Socket.<init>(Socket.java:375)
> > > at java.net.Socket.<init>(Socket.java:189)
> > > at
> > org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > >
> > > Thanks & Regards,
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > For additional commands, e-mail: dev-help@hc.apache.org
> >
> >
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message