From users-return-267000-archive-asf-public=cust-asf.ponee.io@tomcat.apache.org Mon Mar 4 16:20:59 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2A148180627 for ; Mon, 4 Mar 2019 17:20:58 +0100 (CET) Received: (qmail 16839 invoked by uid 500); 4 Mar 2019 16:20:57 -0000 Mailing-List: contact users-help@tomcat.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Tomcat Users List" Delivered-To: mailing list users@tomcat.apache.org Received: (qmail 16828 invoked by uid 99); 4 Mar 2019 16:20:57 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Mar 2019 16:20:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E5BBFC6AC9 for ; Mon, 4 Mar 2019 16:20:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.925 X-Spam-Level: X-Spam-Status: No, score=0.925 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_LOCAL_NOVOWEL=0.5, HK_RANDOM_ENVFROM=0.626, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id eHcfK2KzxSsE for ; Mon, 4 Mar 2019 16:20:54 +0000 (UTC) Received: from mail-io1-f65.google.com (mail-io1-f65.google.com [209.85.166.65]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E98F360E52 for ; Mon, 4 Mar 2019 16:20:53 +0000 (UTC) Received: by mail-io1-f65.google.com with SMTP id x9so4385673iog.12 for ; Mon, 04 Mar 2019 08:20:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-transfer-encoding; bh=gR0Ic8kVTDk7bR3zm3RWjXi73Fm9daZMz3vW6zlvNMM=; b=e/CQAVatQXU2QIsUyvRFuUtli2RQETSU/h+d9AYMiMR1vD4l6K70/kVYXcFBJdaTjX M2raicyuZ+glUKIatI46cOoufAuZhDZHlwiK4pl+9yaV53hZaCDemLLB0hoVdt+nNVnc Bv9J24zDHtNUhS+hQ9ObTJ9cf1B1Pc6UQQzAkI97Ix3OgRepJWQmn2e5fTip76ApTjaZ CNu3PpyvzxJfhGLMTeCmhasC8IVTi8SrHAPrOVN/KvtAHi4RxgxD4C5LGxDFOuHZsTfL i2xRoIYLr/Z4X0lUv1BJT+M70KqlC9ZsHH0O9UfQlDgOIa5Bhe8F+iAUMO5y59Cyvk/v Kk7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-transfer-encoding; bh=gR0Ic8kVTDk7bR3zm3RWjXi73Fm9daZMz3vW6zlvNMM=; b=Ah9vWQWXcd4p8CmzA8zW+cnoy63vJ7QCwjTGXMSSC9emxPOeQ49IGUxHQVGCkZfeaQ nSiXsgufv4P7YOd3DLU87kRpTL4zU/QneenSvSRy9PMvYsitXJ/nuFRTB89Y+tamIW5y 0z1l4/UT4p0anJriOfpcauQXl0WSQq06y/JnsoLfjCzJvVc3nYLeiA7VPtn3jxYoBZ3W El7zrdKqQpf/NZj6dQ3HosQ+eJ9fvIUe/VOgDc6jueQGXFwQKYcoZ0XM8QN79u4DVqaT K3ItE+1vw+kyNE7HLsZKn990CNZp8ZnD2+q1hK5XKyYvbVigxCTrIazXbHHG2/vgRr4y t0Lg== X-Gm-Message-State: APjAAAU+Uyw4+KpL8Z5/l8YSWVk1PQbL6Xg3umH9OngkVxSFF8y1UD2j SgGsPNK9nl4C/Fqxts+gAIdK9cMGz1eoyVJ0sjRHOw== X-Google-Smtp-Source: APXvYqxLXAZiDvqdJNklaYxiwCU8fVb3LgDBHHIbhahnayB8uj7FVqxTrO1P42XyMrab6uqnYQh4z3M4oSJwSy02m/E= X-Received: by 2002:a6b:5908:: with SMTP id n8mr9602440iob.206.1551716452297; Mon, 04 Mar 2019 08:20:52 -0800 (PST) MIME-Version: 1.0 Received: by 2002:a4f:7316:0:0:0:0:0 with HTTP; Mon, 4 Mar 2019 08:20:51 -0800 (PST) In-Reply-To: References: From: John Dale Date: Mon, 4 Mar 2019 09:20:51 -0700 Message-ID: Subject: Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload To: Tomcat Users List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Nice test case. Is the JVM able to garbage collect fast enough? Are you releasing the resources properly in your components as early as possible? Java VM's achieve a steady-state after a certain period of time, but I've found in some cases garbage collection doesn't keep-up with my load tests before steady state is reached. Just an idea. Again - nice test. :) On 3/4/19, Santhosh Kumar wrote: >> Do you actually need all those values and where do they come from in the > first place ? Do you understand what they do ? > > I was just experimenting with all available attributes related to socket > and multiplexing. I have followed the documentations, > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_S= SLHostConfig > >> The buffer sizes seem way >> too large, and maxConcurrentStreamExecution is as well (multiplexing 200 >> threads output over a single socket is not a very good idea usually, >> which >> is why the setting was added). >> The attributes are from java tcp socket attributes, you can check here > > I have given the 10x of default values to handle high load, and > maxConcurrentStreamExecution is also increased assuming it will allow lar= ge > data to be exchanged. I ll set back to default if it doesn't help > > From h2load, the payload of 1000b approx sent with 100 requests with > multiplexing as any, it fails after 65 requests approximately. > > I don't know whether tomcat have any limit set on buffer to restrict data > pushed in a same connection. > > On Mon, Mar 4, 2019 at 7:27 PM R=C3=A9my Maucherat wrot= e: > >> On Mon, Mar 4, 2019 at 10:40 AM Santhosh Kumar >> wrote: >> >> > Hi, >> > >> > We have a tomcat instance which is http2 enabled and it needs to serve >> > large number of requests using multiplexing, so we have configured our >> > instance as follows, >> > >> > > > sslImplementationName=3D"org.apache.tomcat.util.net >> > .openssl.OpenSSLImplementation" >> > protocol=3D"org.apache.coyote.http11.Http11Nio2Protocol" >> > maxThreads=3D"50000" SSLEnabled=3D"true" >> > >> > >> compressibleMimeType=3D"text/html,text/xml,text/plain,text/css,text/java= script,application/javascript,application/json,application/xml" >> > compression=3D"on" minSpareThreads=3D"25" >> > noCompressionUserAgents=3D"gozilla, traviata" scheme=3D"https" >> > secure=3D"true" >> > keystoreFile=3D"conf/myfile.keystore" keystorePass=3D"password" >> > socket.appReadBufSize=3D"81920" >> > socket.appWriteBufSize=3D"81920" socket.rxBufSize=3D"251880" >> > socket.txBufSize=3D"438000"> >> > > > maxConcurrentStreamExecution=3D"200" >> > maxConcurrentStreams=3D"200" >> > className=3D"org.apache.coyote.http2.Http2Protocol"/> >> > >> > >> >> Do you actually need all those values and where do they come from in the >> first place ? Do you understand what they do ? The buffer sizes seem way >> too large, and maxConcurrentStreamExecution is as well (multiplexing 200 >> threads output over a single socket is not a very good idea usually, >> which >> is why the setting was added). >> >> R=C3=A9my >> >> >> > >> > This instance mainly serves concurrent POST request which will have >> payload >> > of size, approx 1000-1500, which can be verified by tomcat logs >> > >> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], >> > Stream >> > [19], Frame type [DATA], Flags [1], Payload size [*1195*] >> > >> > We tested our server with the help of h2load as follows, >> > >> > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt' >> > >> > We are getting this error as follows, >> > >> > >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection >> [0] >> > java.io.IOException: Unable to unwrap data, invalid status >> > [BUFFER_OVERFLOW] >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> > at >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> > at >> > java.base/sun.nio.ch >> > >> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChanne= lImpl.java:552) >> > at >> > java.base/sun.nio.ch >> > >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:2= 76) >> > at >> > java.base/sun.nio.ch >> > >> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:2= 97) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000) >> > at >> > org.apache.tomcat.util.net >> > .SecureNio2Channel.read(SecureNio2Channel.java:1067) >> > at >> > org.apache.tomcat.util.net >> > >> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Ni= o2Endpoint.java:1153) >> > at >> > org.apache.tomcat.util.net >> > .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026) >> > at >> > org.apache.tomcat.util.net >> > .SocketWrapperBase.read(SocketWrapperBase.java:1012) >> > at >> > >> > >> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java= :61) >> > at >> > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69) >> > at >> > >> > >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2Upgrade= Handler.java:322) >> > at >> > >> > >> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2As= yncUpgradeHandler.java:37) >> > at >> > >> > >> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(Upgra= deProcessorInternal.java:54) >> > at >> > >> > >> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.= java:53) >> > at >> > >> > >> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractPro= tocol.java:834) >> > at >> > org.apache.tomcat.util.net >> > .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769) >> > at >> > org.apache.tomcat.util.net >> > .SocketProcessorBase.run(SocketProcessorBase.java:49) >> > at >> > org.apache.tomcat.util.net >> > .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048) >> > at >> > org.apache.tomcat.util.net >> > >> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2C= hannel.java:116) >> > at >> > org.apache.tomcat.util.net >> > >> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2C= hannel.java:109) >> > at >> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) >> > at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158) >> > >> > Why is this error is thrown? How can I configure tomcat to handle >> > concurrent POST requests which have a decent payload? >> > >> > >> > We have tried with various java clients like http-client-5-beta, jetty >> > or >> > okhttp3 and spam requests to our tomcat using http2 multiplexing and w= e >> > found the time taken to process a requests increases(sometimes even >> > 10x) >> > when load is increased. >> > We have tweaked all common configuration related to http2 on both >> > client >> > and server side with no luck. >> > >> > But same tomcat configuration can handle 10s of 1000s of get request >> > concurrently without a problem, its only creating problem with POST >> > requests. >> > >> > What is wrong in our configuration? >> > >> > Kindly someone shed some light. >> > >> > Tomcat - 9.0.16 >> > APR-1.2.18 >> > OpenSSL-1.1.1a >> > JDK-10.0.2 >> > OS - Ubuntu/Centos >> > HeapSize - 4GB >> > RAM -16GB >> > >> > >> > Kindly help >> > >> > -- >> > *With Regards,* >> > *Santhosh Kumar J* >> > >> > > > -- > *With Regards,* > *Santhosh Kumar J* > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org For additional commands, e-mail: users-help@tomcat.apache.org