tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Thomas <ma...@apache.org>
Subject Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Date Thu, 07 Mar 2019 12:47:16 GMT
On 07/03/2019 07:40, Santhosh Kumar wrote:
> From some of the test cases I can safely say that tomcat is hitting some
> limits, I have two test cases ran with two diff size of payload and without
> any queryParams. The servlet is a empty servlet just returns after
> receiving without doing any business side logic

Can you repeat those tests with the NIO connector? It would be helpful
to know if we should be looking at the HTTP/2 code or the low-level
connector I/O code.

Thanks,

Mark


> 
> h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> /home/local/santhosh/A-Test/nghttp2/agentRequest.txt
> https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> starting benchmark...
> spawning thread #0: 1 total client(s). 100 total requests
> TLS Protocol: TLSv1.3
> Cipher: TLS_AES_256_GCM_SHA384
> Server Temp Key: X25519 253 bits
> Application protocol: h2
> progress: 10% done
> progress: 20% done
> progress: 30% done
> progress: 40% done
> progress: 50% done
> 
> finished in 5.16s, 10.48 req/s, 552B/s
> requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46
> errored, 0 timeout
> status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx
> traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings 43.10%),
> 0B (0) data
>                      min         max         mean         sd        +/- sd
> time for request:     1.57ms      9.43ms      2.24ms      1.17ms    94.44%
> time for connect:     4.69ms      4.69ms      4.69ms         0us   100.00%
> time to 1st byte:        0us         0us         0us         0us     0.00%
> req/s           :      10.48       10.48       10.48        0.00   100.00%
> 
> This above configuration always returns 54 succeeded, payload size is 1200B
> (1200x54=64800)
> ------------------------------------------------------------------------------------------------------------------------------
> Now reduce the payload and trying the same test,
> 
> h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt
> https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> starting benchmark...
> spawning thread #0: 1 total client(s). 100 total requests
> TLS Protocol: TLSv1.3
> Cipher: TLS_AES_256_GCM_SHA384
> Server Temp Key: X25519 253 bits
> Application protocol: h2
> progress: 10% done
> progress: 20% done
> progress: 30% done
> progress: 40% done
> progress: 50% done
> progress: 60% done
> progress: 70% done
> progress: 80% done
> 
> finished in 5.21s, 16.11 req/s, 839B/s
> requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16
> errored, 0 timeout
> status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx
> traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings 43.10%),
> 0B (0) data
>                      min         max         mean         sd        +/- sd
> time for request:     1.43ms      5.80ms      2.04ms       760us    89.29%
> time for connect:     5.02ms      5.02ms      5.02ms         0us   100.00%
> time to 1st byte:        0us         0us         0us         0us     0.00%
> req/s           :      16.11       16.11       16.11        0.00   100.00%
> 
> This above configuration always returns 84 succeeded, payload size is 775B
> (775x84=65100)
> ------------------------------------------------------------------------------------------------------------------------------
> Reducing the payload much smaller,
> 
> h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d
> /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt
> https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> starting benchmark...
> spawning thread #0: 1 total client(s). 200 total requests
> TLS Protocol: TLSv1.3
> Cipher: TLS_AES_256_GCM_SHA384
> Server Temp Key: X25519 253 bits
> Application protocol: h2
> progress: 10% done
> progress: 20% done
> progress: 30% done
> progress: 40% done
> progress: 50% done
> progress: 60% done
> progress: 70% done
> progress: 80% done
> progress: 90% done
> 
> finished in 5.41s, 34.40 req/s, 1.73KB/s
> requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14
> errored, 0 timeout
> status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx
> traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings 43.10%),
> 0B (0) data
>                      min         max         mean         sd        +/- sd
> time for request:     1.18ms     13.49ms      1.91ms      1.13ms    95.16%
> time for connect:     5.93ms      5.93ms      5.93ms         0us   100.00%
> time to 1st byte:        0us         0us         0us         0us     0.00%
> req/s           :      34.41       34.41       34.41        0.00   100.00%
> 
> This above configuration always returns 186 succeeded, payload size is
> 356 (356x186=66216)
> 
> On Wed, Mar 6, 2019 at 9:15 PM John Dale <jcdwrrc@gmail.com> wrote:
> 
>> When you run your test(s), does it fail after a certain period of
>> time, or just keep on going under a certain number of requests?
>>
>> Also, to confirm: you're sending 1000 Byte + query strings?
>>
>> Are you doing anything in the server side component to verify that
>> your parameters have been received successfully?
>>
>> I seems very possible that there is increased overhead parsing the
>> request (POST) body.  That's why I was wondering about the dynamics of
>> your test case.  If you can achieve a steady load state, either some
>> optimization of the POST request parser could be done, or you could
>> accept that overhead if it is comparable to other solutions.
>>
>> On 3/6/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>>> I hope so, I used updated packages/components at the time of development.
>>> few may be outdated like tomcat native as I was using 1.2.18 while
>>> developing but 1.2.21 got released recently.
>>>
>>> On Wed, Mar 6, 2019 at 6:18 PM John Dale <jcdwrrc@gmail.com> wrote:
>>>
>>>> Have you upgraded to the most recent release of your major version?
>>>>
>>>> If so, and if this issue still persists, it is something that the core
>>>> development team might want to look at assuming they can replicate the
>>>> issue.
>>>>
>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>>>>> Sometimes more than 10x
>>>>>
>>>>> On Tue, Mar 5, 2019 at 10:00 PM John Dale <jcdwrrc@gmail.com> wrote:
>>>>>
>>>>>> How many orders of magnitude slower are the post requests?
>>>>>>
>>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>>>>>>> I was testing in the localhost
>>>>>>>
>>>>>>> On Tue, Mar 5, 2019 at 9:32 PM John Dale <jcdwrrc@gmail.com>
>> wrote:
>>>>>>>
>>>>>>>> Are you running your test client (h2load) on the same machine,
>> same
>>>>>>>> network, or is it over the net (so, like 20ms latency on
each
>>>>>>>> request)?  The reason I ask is that if you are local (especially),
>>>>>>>> it
>>>>>>>> may queue up too many requests for tomcat to handle in the
testing
>>>>>>>> period with its thread pool.  Will let you know if I have
any
>> other
>>>>>>>> ideas.
>>>>>>>>
>>>>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>>>>>>>>> Bytes
>>>>>>>>>
>>>>>>>>> On Tue, Mar 5, 2019 at 9:28 PM John Dale <jcdwrrc@gmail.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>>> 1000-1500 MB or KB?
>>>>>>>>>>
>>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>>>>>>>>>>> As per the documentation,
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig
>>>>>>>>>>>
>>>>>>>>>>> this connector supports maxPostSize, by default
the limit is
>>>>>>>>>>> set
>>>>>>>>>>> to
>>>>>>>> 2MB
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 5, 2019 at 5:09 AM John Dale <jcdwrrc@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Does anyone know if this connector supports
maxPostSize
>>>>>>>>>>>> parameter?
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have a tomcat instance which is http2
enabled and it
>>>>>>>>>>>>> needs
>>>>>>>>>>>>> to
>>>>>>>>>>>>> serve
>>>>>>>>>>>>> large number of requests using multiplexing,
so we have
>>>>>>>>>>>>> configured
>>>>>>>>>>>>> our
>>>>>>>>>>>>> instance as follows,
>>>>>>>>>>>>>
>>>>>>>>>>>>> <Connector port="9191"  URIEncoding="UTF-8"
>>>>>>>>>>>>> sslImplementationName="org.apache.tomcat.util.net
>>>>>>>>>>>> .openssl.OpenSSLImplementation"
>>>>>>>>>>>>> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>>>>>>>>>>>>>                          maxThreads="50000"
>>>>>>>>>>>>> SSLEnabled="true"
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
>>>>>>>>>>>>>                          compression="on"
>>>> minSpareThreads="25"
>>>>>>>>>>>>> noCompressionUserAgents="gozilla, traviata"
scheme="https"
>>>>>>>>>>>>> secure="true"
>>>>>>>>>>>>> keystoreFile="conf/myfile.keystore"
>> keystorePass="password"
>>>>>>>>>>>>>                          socket.appReadBufSize="81920"
>>>>>>>>>>>>> socket.appWriteBufSize="81920" socket.rxBufSize="251880"
>>>>>>>>>>>>> socket.txBufSize="438000">
>>>>>>>>>>>>>                         <UpgradeProtocol
compression="on"
>>>>>>>>>>>>>
>>>>>> maxConcurrentStreamExecution="200"
>>>>>>>>>>>>> maxConcurrentStreams="200"
>>>>>>>>>>>>> className="org.apache.coyote.http2.Http2Protocol"/>
>>>>>>>>>>>>>       </Connector>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This instance mainly serves concurrent
POST request which
>>>> will
>>>>>>>>>>>>> have
>>>>>>>>>>>> payload
>>>>>>>>>>>>> of size, approx 1000-1500, which can
be verified by tomcat
>>>>>>>>>>>>> logs
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.coyote.http2.Http2Parser.validateFrame
>>>>>>>>>>>>> Connection
>>>>>> [0],
>>>>>>>>>>>>> Stream
>>>>>>>>>>>>> [19], Frame type [DATA], Flags [1], Payload
size [*1195*]
>>>>>>>>>>>>>
>>>>>>>>>>>>> We tested our server with the help of
h2load as follows,
>>>>>>>>>>>>>
>>>>>>>>>>>>> h2load -n100 -c1 -m100 https://localhost:9191/
-d
>>>>>>>>>>>>> '/agentRequest.txt'
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are getting this error as follows,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch
>>>>>>>>>>>>> Connection
>>>>>>>>>>>> [0]
>>>>>>>>>>>>>  java.io.IOException: Unable to unwrap
data, invalid
>> status
>>>>>>>>>>>>> [BUFFER_OVERFLOW]
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.base/sun.nio.ch
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.base/sun.nio.ch
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.base/sun.nio.ch
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .SecureNio2Channel.read(SecureNio2Channel.java:1067)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .SocketWrapperBase.read(SocketWrapperBase.java:1012)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>>>> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.tomcat.util.net
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>>
>> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why is this error is thrown? How can
I configure tomcat to
>>>>>> handle
>>>>>>>>>>>>> concurrent POST requests which have a
decent payload?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have tried with various java clients
like
>>>>>>>>>>>>> http-client-5-beta,
>>>>>>>>>>>>> jetty
>>>>>>>>>>>>> or
>>>>>>>>>>>>> okhttp3 and spam requests to our tomcat
using http2
>>>>>>>>>>>>> multiplexing
>>>>>>>> and
>>>>>>>>>> we
>>>>>>>>>>>>> found the time taken to process a requests
>>>> increases(sometimes
>>>>>>>>>>>>> even
>>>>>>>>>>>>> 10x)
>>>>>>>>>>>>> when load is increased.
>>>>>>>>>>>>> We have tweaked all common configuration
related to http2
>>>>>>>>>>>>> on
>>>>>> both
>>>>>>>>>>>>> client
>>>>>>>>>>>>> and server side with no luck.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But same tomcat configuration can handle
10s of 1000s of
>>>>>>>>>>>>> get
>>>>>>>> request
>>>>>>>>>>>>> concurrently without a problem, its only
creating problem
>>>> with
>>>>>>>>>>>>> POST
>>>>>>>>>>>>> requests.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is wrong in our configuration?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kindly someone shed some light.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tomcat - 9.0.16
>>>>>>>>>>>>> APR-1.2.18
>>>>>>>>>>>>> OpenSSL-1.1.1a
>>>>>>>>>>>>> JDK-10.0.2
>>>>>>>>>>>>> OS - Ubuntu/Centos
>>>>>>>>>>>>> HeapSize - 4GB
>>>>>>>>>>>>> RAM -16GB
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kindly help
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> *With Regards,*
>>>>>>>>>>>>> *Santhosh Kumar J*
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>> users-help@tomcat.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *With Regards,*
>>>>>>>>>>> *Santhosh Kumar J*
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> *With Regards,*
>>>>>>>>> *Santhosh Kumar J*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>> --
>>>>>>> *With Regards,*
>>>>>>> *Santhosh Kumar J*
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *With Regards,*
>>>>> *Santhosh Kumar J*
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>> --
>>> *With Regards,*
>>> *Santhosh Kumar J*
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message