hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sadeep Jayasumana <gayansad...@gmail.com>
Subject Re: HTTPCore-NIO: Server rejecting socket connections?
Date Sat, 28 Jan 2012 18:16:11 GMT
Hi,

On Fri, Jan 27, 2012 at 6:50 PM, Oleg Kalnichevski <olegk@apache.org> wrote:

> On Fri, 2012-01-27 at 15:18 +0530, Sadeep Jayasumana wrote:
> > Hi,
> >
> > I have written an HTTP server using HTTPCore-NIO and currently testing it
> > with httpcore-ab [1]. I have been seeing very good TPS values so far. But
> > when I try to send 5KB HTTP messages with 2500 and concurrency level, 50
> > messages per thread, I'm getting the following exception several times
> from
> > httpcore-ab and it results in "Failed requests" in the httpcore-ab
> output.
> > No exceptions are displayed from the server side. Hence, I think the
> server
> > is rejecting some socket connections because it's already fully loaded.
> >
> > I'm running the server and load generator on two server machines that are
> > connected with Gigabit Ethernet. Each server has 16 CPU cores, 16 GB
> memory
> > etc.
> >
> > I have increased the socket timeout in both httpcore-ab and server side
> > ListeningIOReactor to 360000. 16 threads are allocated for the reactor.
> > Messages received by the server are processed in a separate worker pool
> > which has ~500 threads. I have also increased the open file limit of my
> OS
> > (Debian) to 65535.
> >
> > Is there any tuning parameter of HTTPCore that I can used to get rid of
> > this exception? Perhaps, something similar to "acceptQueueSize" in Jetty
> >  [2]?
> >
>
> Hi Sadeep
>
> A couple of points in no particular logical order.
>
> (1) HttpCore does not depend on a logging toolkit and does not do any
> logging per default. If it ever encounters an abnormal situation,
> instead of logging such an event it throws an exception and terminates.
> Quite often this is not desirable, so one can provide an exception
> handler that can log using a a logging toolkit of choice or handle a
> particular type of exception in an application specific way.
>
> If the server did not terminate under with an exception, I kind of
> suspect it was still trying to chew through the load without
> encountering anything abnormal. Unless you have added a custom exception
> handler.
>
> (2) HttpCore does not have a concept of accepted connection queue. As
> soon as the listener accepts an incoming connection its channel gets
> immediately added to the underlying i/o selector. No intermediate
> queuing takes place.


> (3) Given that the connection times out on the client, this sounds more
> like a client side issue. Was connection really suck in
> Socket#socketConnect for 360000?
>

I'm using httpcore-ab [1] as the client. I have set socket timeout of using
-t 360000, and changed the connection timeout with the patch shown below.
But I'm starting to get this error well before 360000 millis past the test
startup. Therefore, I there is no possibility that the connection attempt
hangs for 360000 millis before printing this exception. I was thinking
whether there are upper bounds for these limits that cause my settings to
be neglected.

--- src/main/java/org/apache/http/benchmark/HttpBenchmark.java (revision
1205292)
+++ src/main/java/org/apache/http/benchmark/HttpBenchmark.java (working
copy)
@@ -227,7 +227,8 @@
             .setParameter(HttpProtocolParams.USER_AGENT, "HttpCore-AB/1.1")
             .setBooleanParameter(HttpProtocolParams.USE_EXPECT_CONTINUE,
useExpectContinue)

 .setBooleanParameter(HttpConnectionParams.STALE_CONNECTION_CHECK, false)
-            .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
socketTimeout);
+            .setIntParameter(HttpConnectionParams.SO_TIMEOUT,
socketTimeout)
+            .setIntParameter(HttpConnectionParams.CONNECTION_TIMEOUT, 360
* 1000);
         return params;
     }
 }


>
> (4) Quite often it is useful to see what it really going on on the I/O
> reactor level. You might want to add I/O event and wire logging by
> copying logging components from the contrib package [1]
>

I could do this. But will wire logging really help at this concurrency
level and messages/thread?

[1]
https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab

Thanks,
Sadeep

>
> I hope this helps somewhat. In any way I can say I am quite sure
> HttpCore does not reject incoming connections no matter what.


> Oleg
>
> [1]
>
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/branches/4.1.x/httpcore-contrib/src/main/java/org/apache/http/contrib/logging/
>
> > [1]
> >
> https://svn.apache.org/repos/asf/httpcomponents/httpcore/tags/4.1.3/httpcore-ab
> > [2] http://docs.codehaus.org/display/JETTY/Configuring+Connectors
> > [3]
> > java.net.ConnectException: Connection timed out
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > at java.net.Socket.connect(Socket.java:529)
> > at java.net.Socket.connect(Socket.java:478)
> > at java.net.Socket.<init>(Socket.java:375)
> > at java.net.Socket.<init>(Socket.java:189)
> > at
> org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > java.net.ConnectException: Connection timed out
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> > at java.net.Socket.connect(Socket.java:529)
> > at java.net.Socket.connect(Socket.java:478)
> > at java.net.Socket.<init>(Socket.java:375)
> > at java.net.Socket.<init>(Socket.java:189)
> > at
> org.apache.http.benchmark.BenchmarkWorker.run(BenchmarkWorker.java:207)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > Thanks & Regards,
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>


-- 

Sadeep Jayasumana****

Email: gayansadeep@gmail.com****

Phone: +94-77-2266507

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message