tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Santhosh Kumar <santhosh...@gmail.com>
Subject Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Date Mon, 04 Mar 2019 09:39:46 GMT
Hi,

We have a tomcat instance which is http2 enabled and it needs to serve
large number of requests using multiplexing, so we have configured our
instance as follows,

<Connector port="9191"  URIEncoding="UTF-8"
sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
                         maxThreads="50000" SSLEnabled="true"
compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
                         compression="on" minSpareThreads="25"
noCompressionUserAgents="gozilla, traviata" scheme="https" secure="true"
keystoreFile="conf/myfile.keystore" keystorePass="password"
                         socket.appReadBufSize="81920"
socket.appWriteBufSize="81920" socket.rxBufSize="251880"
socket.txBufSize="438000">
                        <UpgradeProtocol compression="on"
                               maxConcurrentStreamExecution="200"
maxConcurrentStreams="200"
className="org.apache.coyote.http2.Http2Protocol"/>
      </Connector>

This instance mainly serves concurrent POST request which will have payload
of size, approx 1000-1500, which can be verified by tomcat logs

org.apache.coyote.http2.Http2Parser.validateFrame Connection [0], Stream
[19], Frame type [DATA], Flags [1], Payload size [*1195*]

We tested our server with the help of h2load as follows,

h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt'

We are getting this error as follows,


org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection [0]
 java.io.IOException: Unable to unwrap data, invalid status
[BUFFER_OVERFLOW]
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
        at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
        at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
        at
java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
        at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
        at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
        at
org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1067)
        at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
        at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
        at
org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1012)
        at
org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
        at
org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
        at
org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
        at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
        at
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
        at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
        at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
        at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
        at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)

Why is this error is thrown? How can I configure tomcat to handle
concurrent POST requests which have a decent payload?


We have tried with various java clients like http-client-5-beta, jetty or
okhttp3 and spam requests to our tomcat using http2 multiplexing and we
found the time taken to process a requests increases(sometimes even 10x)
when load is increased.
We have tweaked all common configuration related to http2 on both client
and server side with no luck.

But same tomcat configuration can handle 10s of 1000s of get request
concurrently without a problem, its only creating problem with POST
requests.

What is wrong in our configuration?

Kindly someone shed some light.

Tomcat - 9.0.16
APR-1.2.18
OpenSSL-1.1.1a
JDK-10.0.2
OS - Ubuntu/Centos
HeapSize - 4GB
RAM -16GB


Kindly help

-- 
*With Regards,*
*Santhosh Kumar J*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message