tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Dale <jcdw...@gmail.com>
Subject Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Date Mon, 04 Mar 2019 16:20:51 GMT
Nice test case.  Is the JVM able to garbage collect fast enough?  Are
you releasing the resources properly in your components as early as
possible?  Java VM's achieve a steady-state after a certain period of
time, but I've found in some cases garbage collection doesn't keep-up
with my load tests before steady state is reached.  Just an idea.
Again - nice test. :)

On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>> Do you actually need all those values and where do they come from in the
> first place ? Do you understand what they do ?
>
> I was just experimenting with all available attributes related to socket
> and multiplexing. I have followed the documentations,
> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig
>
>> The buffer sizes seem way
>> too large, and maxConcurrentStreamExecution is as well (multiplexing 200
>> threads output over a single socket is not a very good idea usually,
>> which
>> is why the setting was added).
>> The attributes are from java tcp socket attributes, you can check here
>
> I have given the 10x of default values to handle high load, and
> maxConcurrentStreamExecution is also increased assuming it will allow large
> data to be exchanged. I ll set back to default if it doesn't help
>
> From h2load, the payload of 1000b approx sent with 100 requests with
> multiplexing as any, it fails after 65 requests approximately.
>
> I don't know whether tomcat have any limit set on buffer to restrict data
> pushed in a same connection.
>
> On Mon, Mar 4, 2019 at 7:27 PM Rémy Maucherat <remm@apache.org> wrote:
>
>> On Mon, Mar 4, 2019 at 10:40 AM Santhosh Kumar <santhosh89j@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > We have a tomcat instance which is http2 enabled and it needs to serve
>> > large number of requests using multiplexing, so we have configured our
>> > instance as follows,
>> >
>> > <Connector port="9191"  URIEncoding="UTF-8"
>> > sslImplementationName="org.apache.tomcat.util.net
>> > .openssl.OpenSSLImplementation"
>> > protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>> >                          maxThreads="50000" SSLEnabled="true"
>> >
>> >
>> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
>> >                          compression="on" minSpareThreads="25"
>> > noCompressionUserAgents="gozilla, traviata" scheme="https"
>> > secure="true"
>> > keystoreFile="conf/myfile.keystore" keystorePass="password"
>> >                          socket.appReadBufSize="81920"
>> > socket.appWriteBufSize="81920" socket.rxBufSize="251880"
>> > socket.txBufSize="438000">
>> >                         <UpgradeProtocol compression="on"
>> >                                maxConcurrentStreamExecution="200"
>> > maxConcurrentStreams="200"
>> > className="org.apache.coyote.http2.Http2Protocol"/>
>> >       </Connector>
>> >
>>
>> Do you actually need all those values and where do they come from in the
>> first place ? Do you understand what they do ? The buffer sizes seem way
>> too large, and maxConcurrentStreamExecution is as well (multiplexing 200
>> threads output over a single socket is not a very good idea usually,
>> which
>> is why the setting was added).
>>
>> Rémy
>>
>>
>> >
>> > This instance mainly serves concurrent POST request which will have
>> payload
>> > of size, approx 1000-1500, which can be verified by tomcat logs
>> >
>> > org.apache.coyote.http2.Http2Parser.validateFrame Connection [0],
>> > Stream
>> > [19], Frame type [DATA], Flags [1], Payload size [*1195*]
>> >
>> > We tested our server with the help of h2load as follows,
>> >
>> > h2load -n100 -c1 -m100 https://localhost:9191/ -d '/agentRequest.txt'
>> >
>> > We are getting this error as follows,
>> >
>> >
>> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Connection
>> [0]
>> >  java.io.IOException: Unable to unwrap data, invalid status
>> > [BUFFER_OVERFLOW]
>> >         at
>> > org.apache.tomcat.util.net
>> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
>> >         at
>> > org.apache.tomcat.util.net
>> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>> >         at
>> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>> >         at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>> >         at
>> > java.base/sun.nio.ch
>> >
>> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
>> >         at
>> > java.base/sun.nio.ch
>> >
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>> >         at
>> > java.base/sun.nio.ch
>> >
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>> >         at
>> > org.apache.tomcat.util.net
>> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
>> >         at
>> > org.apache.tomcat.util.net
>> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>> >         at
>> > org.apache.tomcat.util.net
>> > .SecureNio2Channel.read(SecureNio2Channel.java:1067)
>> >         at
>> > org.apache.tomcat.util.net
>> >
>> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
>> >         at
>> > org.apache.tomcat.util.net
>> > .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
>> >         at
>> > org.apache.tomcat.util.net
>> > .SocketWrapperBase.read(SocketWrapperBase.java:1012)
>> >         at
>> >
>> >
>> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
>> >         at
>> > org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
>> >         at
>> >
>> >
>> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
>> >         at
>> >
>> >
>> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
>> >         at
>> >
>> >
>> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
>> >         at
>> >
>> >
>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
>> >         at
>> >
>> >
>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
>> >         at
>> > org.apache.tomcat.util.net
>> > .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
>> >         at
>> > org.apache.tomcat.util.net
>> > .SocketProcessorBase.run(SocketProcessorBase.java:49)
>> >         at
>> > org.apache.tomcat.util.net
>> > .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
>> >         at
>> > org.apache.tomcat.util.net
>> >
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
>> >         at
>> > org.apache.tomcat.util.net
>> >
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
>> >         at
>> > java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>> >         at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>> >
>> > Why is this error is thrown? How can I configure tomcat to handle
>> > concurrent POST requests which have a decent payload?
>> >
>> >
>> > We have tried with various java clients like http-client-5-beta, jetty
>> > or
>> > okhttp3 and spam requests to our tomcat using http2 multiplexing and we
>> > found the time taken to process a requests increases(sometimes even
>> > 10x)
>> > when load is increased.
>> > We have tweaked all common configuration related to http2 on both
>> > client
>> > and server side with no luck.
>> >
>> > But same tomcat configuration can handle 10s of 1000s of get request
>> > concurrently without a problem, its only creating problem with POST
>> > requests.
>> >
>> > What is wrong in our configuration?
>> >
>> > Kindly someone shed some light.
>> >
>> > Tomcat - 9.0.16
>> > APR-1.2.18
>> > OpenSSL-1.1.1a
>> > JDK-10.0.2
>> > OS - Ubuntu/Centos
>> > HeapSize - 4GB
>> > RAM -16GB
>> >
>> >
>> > Kindly help
>> >
>> > --
>> > *With Regards,*
>> > *Santhosh Kumar J*
>> >
>>
>
>
> --
> *With Regards,*
> *Santhosh Kumar J*
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message