Return-Path: Delivered-To: apmail-jakarta-tomcat-dev-archive@jakarta.apache.org Received: (qmail 79213 invoked by uid 500); 14 Jun 2001 14:31:32 -0000 Mailing-List: contact tomcat-dev-help@jakarta.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: list-post: Reply-To: tomcat-dev@jakarta.apache.org Delivered-To: mailing list tomcat-dev@jakarta.apache.org Received: (qmail 79189 invoked from network); 14 Jun 2001 14:31:30 -0000 Message-id: <23018738@mailbox2.Hitchcock.ORG> Date: 14 Jun 2001 10:31:30 EDT From: doug@hornig.net (Douglas E. Hornig) Subject: Re: Solaris Performance Problem To: tomcat-dev@jakarta.apache.org MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Spam-Rating: h31.sny.collab.net 1.6.2 0/1000/N For those of you who remember my question from several weeks ago = regarding a response time delay between tomcat running standalone = on Solaris and a Windows client. I modified my own tomcat 3.2.2 as = Gal suggested and the delay is gone. I'll leave this to the pros to decide how to incorporate this into = tomcat properly. Thanks to everyone who helped me out with this problem (especially = Gal!) Doug --- Forwarded message from Gal Shalif - Sun Israel - Software = Engineer --- >From: Gal Shalif - Sun Israel - Software Engineer = >Date: Sun, 10 Jun 2001 14:47:44 +0300 (IDT) >To: "Douglas E. Hornig" >Subject: Re: Solaris Performance Problem Hello, It should be fixed with a call to Socket.setTcpNoDelay() on the = Tomcat server socket. A C program that I wrote had the same problem as the Tomcat server. and the problem was fixed with the setting of the TCP NODELAY transmission. Anybody there who volunteer to add this option to the Tomcat Java server code? My analysis of the problem is attached at the end of this mail. > >From: doug@hornig.net (Douglas E. Hornig) > Subject: Solaris Performance Problem > To: tomcat-user@jakarta.apache.org >=20 > I have run into a strance performance problem with Tomcat that I = am at a loss to figure out. In a nutshell, when I run Tomcat = (version 3.2.1, standalone) on a Solaris Sparc server and hit it = with a Windows client, the minimum response time is is abount 0.2 = seconds. This is for a simple servlet that just returns some = static data. That may not sound like a lot of time but if a client = makes several requests to the server it can add up fast. If I run = the server on a Linux PC that number is about 0.01 seconds or less. = I also get good performce if I use a Linux client and hit the = Solaris server. >=20 > I have tried using different PC clients (different PCs, one = running NT 4 and one with Windows 2000), different test programs = (one in Java and one in Visual Basic) and it makes no difference. >=20 > I have tried using different Solaris servers, a 420R and an Ultra = 5, neither with any kind of load. I have tried different JDKs = (1.2.2 and 1.3.0). No difference. >=20 > I wrote a quickie standalone Java server for the Sparcs and their = response time was very fast (< 0.01 secs). The server reads a = request, sends a reply, and closes the socket much like a web = server would. >=20 > So the problem combination seems to be Solaris server with Tomcat = and a PC client. I know this sounds nuts but it's totally = repeatable and a very serious problem. Does anyone have an idea = what I can do to get to the bottom of this? >=20 > Thanks a lot. >=20 > Douglas Hornig > Dartmouth-Hitchcock Medical Center > Lebanon, NH ---------------------------------------------------- slow response time with Solaris 8 server and=20 Windows 2000 TCP client as tested with Tomcat and IE ---------------------------------------------------- Problem description: -------------------- Tomcat Hello World examples (/examples/servlet/HelloWorldExample)=20 divide the response into two parts: HTTP protocol header HTML data =20 At the Solaris TCP level, the second part of the response will = not be send till the Windows client acknowledge (ACK) the first part. The result: a delay of ~170ms (dependent on client/server = configuration) =20 The configuration in problem: ----------------------------- Solaris 8 running Tomcat 3.2.1 server Windows 2000 client machine running a web client And the raw data: ----------------- The output was gathered with: truss -fdDla -o /tmp/tmp.dat -p 5732 On a Solaris machine: SunOS node2mde 5.8 Generic_108528-05 sun4u sparc = SUNW,Ultra-60 root nexus =3D Sun Enterprise 220R (2 X UltraSPARC-II = 450MHz) Tomcat server (on the Solaris 8) send two sends within a few = milliseconds: time stamp duration system-call and its arguments (seconds) (seconds) 100759.3625 0.0009 send(14, "HTTP/1.0 200" ...., 179, = 0) =3D 179 100759.3629 0.0004 send(14, "\n, and ----------- is set with setsockopt(3SOCKET) and tested with getsockopt(3SOCKET). The option level for the setsockopt() call is the protocol number for TCP, available from getprotobyname(3SOCKET). And Java documentation about TCP_NODELAY (from the Java 1.3.1 = docs): --------------------------------------------------------------------= http://java.sun.com/j2se/1.3/docs/api/java/net/SocketOptions.html TCP_NODELAY =20 public static final int TCP_NODELAY =20 Disable Nagle's algorithm for this connection. Written = data to the network is not buffered pending acknowledgment of = previously written data. =20 Valid for TCP only: SocketImpl.=20 =20 See Also:=20 Socket.setTcpNoDelay(boolean), = Socket.getTcpNoDelay() APPENDIX A: Win200 ie client and Solaris Tomcat Server ------------------------------------------------------ Tomcat on Solaris 8 server (Ultra10) and a Win200 client = (laptop). A 200ms delay beween the Solaris send of the HTTP header and the Win200 acknoledge: the Win200 client ACK the first Tomcat transfer (HTTP header - HTTP/1.0 200 OK ...) with 200ms delay. Here is a snoop of the transfers: --------------------------------- capture: -------- # snoop -o /tmp/ariel.snoop -d hme0 host ariel =20 summary: -------- # snoop -t a -S -i /tmp/ariel.snoop '(host ariel and host = galfield)' 1 16:32:45.02819 ariel -> galfield length: 62 = HTTP (proxy) C port=3D2076=20 2 16:32:45.02822 galfield -> ariel length: 62 = HTTP (proxy) R port=3D2076=20 3 16:32:45.02847 ariel -> galfield length: 60 = HTTP (proxy) C port=3D2076=20 4 16:32:45.02933 ariel -> galfield length: 420 = HTTP GET /examples/servlet/HelloWorldExample HTTP/1.1 5 16:32:45.02943 galfield -> ariel length: 54 = HTTP (proxy) R port=3D2076=20 6 16:32:45.03498 galfield -> ariel length: 226 = HTTP HTTP/1.0 200 OK -> 7 16:32:45.23190 ariel -> galfield length: 60 = HTTP (proxy) C port=3D2076=20 8 16:32:45.23194 galfield -> ariel length: 454 = HTTP (body) 9 16:32:45.23249 ariel -> galfield length: 60 = HTTP (proxy) C port=3D2076=20 10 16:32:45.24947 ariel -> galfield length: 60 = HTTP (proxy) C port=3D2076=20 11 16:32:45.24950 galfield -> ariel length: 54 = HTTP (proxy) R port=3D2076=20 #=20 =20 APPENDIX B: NT netscape client and Solaris Tomcat Server -------------------------------------------------------- Simmilar to Win200 ie client. # snoop -t a -S -i /tmp/karish.snoop 1 19:24:10.87380 karish -> galfield length: 60 = HTTP (proxy) C port=3D1469=20 2 19:24:10.87384 galfield -> karish length: 58 = HTTP (proxy) R port=3D1469=20 3 19:24:10.87402 karish -> galfield length: 60 = HTTP (proxy) C port=3D1469=20 4 19:24:10.87735 karish -> galfield length: 373 = HTTP GET /examples/servlet/HelloWorldExample HTTP/1.0 5 19:24:10.87745 galfield -> karish length: 54 = HTTP (proxy) R port=3D1469=20 6 19:24:10.88701 galfield -> karish length: 226 = HTTP HTTP/1.0 200 OK -> 7 19:24:11.03993 karish -> galfield length: 60 = HTTP (proxy) C port=3D1469=20 8 19:24:11.03996 galfield -> karish length: 454 = HTTP (body) 9 19:24:11.04024 karish -> galfield length: 60 = HTTP (proxy) C port=3D1469=20 10 19:24:11.09155 karish -> galfield length: 60 = HTTP (proxy) C port=3D1469=20 11 19:24:11.09158 galfield -> karish length: 54 = HTTP (proxy) R port=3D1469=20 APPENDIX C: Solaris netscape client and Solaris Tomcat Server ------------------------------------------------------------- Tomcat on Solaris 8 server (Ultra10) and a Solaris 2.6 client = (Ultra1). The delay does not exist: the Solaris client ACK the first Tomcat transfer (HTTP header - HTTP/1.0 200 OK ...) without delay and = the=20 second transfer (HTML body) follow emediately. The output from snoop proof it. Here is a snoop of the transfers: --------------------------------- capture: -------- # snoop -o /tmp/galfield.snoop -d hme0 host emek =20 summary: -------- # snoop -t a -S -i /tmp/galfield.snoop '(host emek and host = galfield)' 1 18:29:7.79029 emek -> galfield length: 60 = HTTP (proxy) C port=3D50570=20 2 18:29:7.79098 galfield -> emek length: 58 = HTTP (proxy) R port=3D50570=20 3 18:29:7.79133 emek -> galfield length: 60 = HTTP (proxy) C port=3D50570=20 4 18:29:7.79853 emek -> galfield length: 461 = HTTP GET /examples/servlet/HelloWorldExample HTTP/1.0 5 18:29:7.79864 galfield -> emek length: 54 = HTTP (proxy) R port=3D50570=20 6 18:29:7.80270 galfield -> emek length: 226 = HTTP HTTP/1.0 200 OK -> 7 18:29:7.80353 emek -> galfield length: 60 = HTTP (proxy) C port=3D50570=20 8 18:29:7.80356 galfield -> emek length: 454 = HTTP (body) 9 18:29:7.81623 emek -> galfield length: 60 = HTTP (proxy) C port=3D50570=20 10 18:29:7.87945 emek -> galfield length: 60 = HTTP (proxy) C port=3D50570=20 11 18:29:7.87948 galfield -> emek length: 54 = HTTP (proxy) R port=3D50570=20 #=20 =20 APPENDIX D: Windows NT netscape client and Solaris simple web = Server --------------------------------------------------------------------= The delay exist with the default TCP options (TCP_NODELAY is 0) but disapear when the server socket option TCP_NODELAY is set to = 1: int just_say_no =3D nodelay; setsockopt(s, IPPROTO_TCP, TCP_NODELAY, (void *)&just_say_no, = sizeof(int)); Here is a snoop of the transfers: --------------------------------- with the NODELAY on (default setting): -------------------------------------- 1 12:06:6.56395 karish -> galfield length: 60 = INGRESLOCK R port=3D1102=20 2 12:06:6.56399 galfield -> karish length: 58 = INGRESLOCK C port=3D1102=20 3 12:06:6.56417 karish -> galfield length: 60 = INGRESLOCK R port=3D1102=20 4 12:06:6.57547 karish -> galfield length: 355 = INGRESLOCK R port=3D1102 GET /examples/servle 5 12:06:6.57557 galfield -> karish length: 54 = INGRESLOCK C port=3D1102=20 6 12:06:6.57579 galfield -> karish length: 226 = INGRESLOCK C port=3D1102 HTTP/1.0 200 OK\r\nCon -> 7 12:06:6.68511 karish -> galfield length: 60 = INGRESLOCK R port=3D1102=20 8 12:06:6.68514 galfield -> karish length: 448 = INGRESLOCK C port=3D1102 \n\n galfield length: 60 = INGRESLOCK R port=3D1102=20 10 12:06:6.71675 karish -> galfield length: 60 = INGRESLOCK R port=3D1102=20 11 12:06:6.71677 galfield -> karish length: 54 = INGRESLOCK C port=3D1102=20 with the NODELAY off: --------------------- 1 12:09:51.75895 karish -> galfield length: 60 = TCP D=3D1102 S=3D1526 Syn Seq=3D2670688616 Len=3D0 Win=3D8192 = Options=3D<mss 1460> 2 12:09:51.75899 galfield -> karish length: 58 = TCP D=3D1526 S=3D1102 Syn Ack=3D2670688617 Seq=3D197375344 Len=3D0 = Win=3D24820 Options=3D<mss 1460> 3 12:09:51.75917 karish -> galfield length: 60 = TCP D=3D1102 S=3D1526 Ack=3D197375345 Seq=3D2670688617 Len=3D0 = Win=3D8760 4 12:09:51.76217 karish -> galfield length: 373 = TCP D=3D1102 S=3D1526 Ack=3D197375345 Seq=3D2670688617 Len= =3D319 Win=3D8760 5 12:09:51.77097 galfield -> karish length: 54 = TCP D=3D1526 S=3D1102 Ack=3D2670688936 Seq=3D197375345 Len=3D0 = Win=3D24820 6 12:09:51.77123 galfield -> karish length: 226 = TCP D=3D1526 S=3D1102 Ack=3D2670688936 Seq=3D197375345 Len= =3D172 Win=3D24820 7 12:09:51.77131 galfield -> karish length: 448 = TCP D=3D1526 S=3D1102 Ack=3D2670688936 Seq=3D197375517 Len= =3D394 Win=3D24820 8 12:09:51.77147 galfield -> karish length: 54 = TCP D=3D1526 S=3D1102 Fin Ack=3D2670688936 Seq=3D197375911 Len=3D0 = Win=3D24820 9 12:09:51.77155 karish -> galfield length: 60 = TCP D=3D1102 S=3D1526 Ack=3D197375911 Seq=3D2670688936 Len=3D0 = Win=3D8194 10 12:09:51.77160 karish -> galfield length: 60 = TCP D=3D1102 S=3D1526 Ack=3D197375912 Seq=3D2670688936 Len=3D0 = Win=3D8194 11 12:09:51.81374 karish -> galfield length: 60 = TCP D=3D1102 S=3D1526 Fin Ack=3D197375912 Seq=3D2670688936 Len=3D0 = Win=3D8194 12 12:09:51.81377 galfield -> karish length: 54 = TCP D=3D1526 S=3D1102 Ack=3D2670688937 Seq=3D197375912 Len=3D0 = Win=3D24820 Other Resources: ---------------- * Apache setting of the server socket options: http://httpd.apache.org/dist/apache_1.3.19.tar.gz /src/main/http_main.c