tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Santhosh Kumar <santhosh...@gmail.com>
Subject Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Date Thu, 07 Mar 2019 14:09:10 GMT
I have executed few test cases in both NIO and NIO2 as Mark has requested
for testing NIO connector.

------------------------------------------Below is
NIO------------------------------------------------

h2load -n50 -c1 -m1 --header="Content-Type:application/json" -d
/home/local/santhosh/A-Test/nghttp2/agentRequest.txt
https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
starting benchmark...
spawning thread #0: 1 total client(s). 50 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 133.78ms, 373.74 req/s, 18.91KB/s
requests: 50 total, 50 started, 50 done, 50 succeeded, 0 failed, 0 errored,
0 timeout
status codes: 50 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 2.53KB (2591) total, 1.61KB (1650) headers (space savings 43.10%),
0B (0) data
                     min         max         mean         sd        +/- sd
time for request:     1.30ms      6.49ms      2.53ms      1.37ms    80.00%
time for connect:     3.86ms      3.86ms      3.86ms         0us   100.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :     375.11      375.11      375.11        0.00   100.00%

This is for 1200B of data

------------------------------------------Below is
NIO------------------------------------------------

h2load -n80 -c1 -m1 --header="Content-Type:application/json" -d
/home/local/santhosh/A-Test/nghttp2/agentRequest2.txt
https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
starting benchmark...
spawning thread #0: 1 total client(s). 80 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 153.28ms, 521.93 req/s, 26.26KB/s
requests: 80 total, 80 started, 80 done, 80 succeeded, 0 failed, 0 errored,
0 timeout
status codes: 80 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 4.02KB (4121) total, 2.58KB (2640) headers (space savings 43.10%),
0B (0) data
                     min         max         mean         sd        +/- sd
time for request:     1.12ms      5.17ms      1.83ms       657us    86.25%
time for connect:     4.84ms      4.84ms      4.84ms         0us   100.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :     523.06      523.06      523.06        0.00   100.00%

This is for 775B of data

------------------------------------------Below is
NIO------------------------------------------------

h2load -n180 -c1 -m1 --header="Content-Type:application/json" -d
/home/local/santhosh/A-Test/nghttp2/agentRequest3.txt
https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
starting benchmark...
spawning thread #0: 1 total client(s). 180 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 280.89ms, 640.82 req/s, 32.06KB/s
requests: 180 total, 180 started, 180 done, 180 succeeded, 0 failed, 0
errored, 0 timeout
status codes: 180 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 9.00KB (9221) total, 5.80KB (5940) headers (space savings 43.10%),
0B (0) data
                     min         max         mean         sd        +/- sd
time for request:     1.05ms      6.15ms      1.50ms       637us    91.11%
time for connect:     7.24ms      7.24ms      7.24ms         0us   100.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :     641.64      641.64      641.64        0.00   100.00%

This is for 352B of data


------------------------------------------Below is
NIO------------------------------------------------

Above are the Ideal cases it works for NIO beyond which it breaks.

For ex: take the case 1 and increase the request count from 50 to 60 it
hangs and its applicable for all the 3 cases , tweak the request count
beyond the configured 50(1200B), 80(775B) or 180(352B) the test will hang
forever

h2load -n60 -c1 -m1 --header="Content-Type:application/json" -d
/home/local/santhosh/A-Test/nghttp2/agentRequest.txt
https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
starting benchmark...
spawning thread #0: 1 total client(s). 60 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done

I don see any exception in catalina logs unless I interrupt the hanging
test,

 java.io.EOFException
        at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1206)
        at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1177)
        at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1108)
        at
org.apache.coyote.http2.Http2UpgradeHandler.fill(Http2UpgradeHandler.java:1229)
        at
org.apache.coyote.http2.Http2Parser$Input.fill(Http2Parser.java:692)
        at
org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:76)
        at
org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
        at
org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
        at
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

p.s: This exception is only thrown when I interrupt the hanging test,
otherwise its not thrown

------------------------------------------Below is
NIO2------------------------------------------------

The same testcase ran in NIO2, will return response even though the test
has failures, like below

h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
/home/local/santhosh/A-Test/nghttp2/agentRequest.txt
https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
starting benchmark...
spawning thread #0: 1 total client(s). 100 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done

finished in 5.40s, 10.01 req/s, 527B/s
requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46
errored, 0 timeout
status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings 43.10%),
0B (0) data
                     min         max         mean         sd        +/- sd
time for request:     2.26ms    147.52ms      6.37ms     19.60ms    98.15%
time for connect:    25.52ms     25.52ms     25.52ms         0us   100.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :      10.01       10.01       10.01        0.00   100.00%


This is for 1200B of data
------------------------------------------Below is
NIO2------------------------------------------------

The above test case was tested with various multiplexing numbers with same
1200B of payload, it was working till multiplexing reaches 25 beyond which
it failed immediately. If I reduce the payload, multiplexing is slightly
increased from 25 to 30/35 and beyond which it again fails.


For ex:

h2load -n100 -c1 -m30 --header="Content-Type:application/json" -d
/home/local/santhosh/A-Test/nghttp2/agentRequest.txt
https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
starting benchmark...
spawning thread #0: 1 total client(s). 100 total requests
TLS Protocol: TLSv1.3
Cipher: TLS_AES_256_GCM_SHA384
Server Temp Key: X25519 253 bits
Application protocol: h2

finished in 72.08ms, 0.00 req/s, 568B/s
requests: 100 total, 30 started, 0 done, 0 succeeded, 100 failed, 100
errored, 0 timeout
status codes: 0 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 41B (41) total, 0B (0) headers (space savings 0.00%), 0B (0) data
                     min         max         mean         sd        +/- sd
time for request:        0us         0us         0us         0us     0.00%
time for connect:     9.32ms      9.32ms      9.32ms         0us   100.00%
time to 1st byte:        0us         0us         0us         0us     0.00%
req/s           :       0.00        0.00        0.00        0.00   100.00%

The exception is as follows,

java.io.IOException: Unable to unwrap data, invalid status [BUFFER_OVERFLOW]
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
        at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
        at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
        at
sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
        at
sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
        at
sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
        at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
        at
org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1067)
        at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
        at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
        at
org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1012)
        at
org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
        at
org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
        at
org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
        at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
        at
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
        at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
        at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
        at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
        at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
        at
sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
        at
sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
        at
sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
        at
org.apache.tomcat.util.net.SecureNio2Channel.handshakeInternal(SecureNio2Channel.java:297)
        at
org.apache.tomcat.util.net.SecureNio2Channel.handshake(SecureNio2Channel.java:216)
        at
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1747)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
        at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:99)
        at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:92)
        at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
        at sun.nio.ch.Invoker$2.run(Invoker.java:218)
        at
sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)


On Thu, Mar 7, 2019 at 7:24 PM Rémy Maucherat <remm@apache.org> wrote:

> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas <markt@apache.org> wrote:
>
> > On 07/03/2019 07:40, Santhosh Kumar wrote:
> > > From some of the test cases I can safely say that tomcat is hitting
> some
> > > limits, I have two test cases ran with two diff size of payload and
> > without
> > > any queryParams. The servlet is a empty servlet just returns after
> > > receiving without doing any business side logic
> >
> > Can you repeat those tests with the NIO connector? It would be helpful
> > to know if we should be looking at the HTTP/2 code or the low-level
> > connector I/O code.
> >
>
> I was planning to investigate since I'm hunting NIO2 additional issues
> after the fix for BZ63182. This one looks simpler to reproduce at least
> [assuming there's an issue].
>
> Rémy
>
>
> >
> > Thanks,
> >
> > Mark
> >
> >
> > >
> > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> > > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt
> > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > > starting benchmark...
> > > spawning thread #0: 1 total client(s). 100 total requests
> > > TLS Protocol: TLSv1.3
> > > Cipher: TLS_AES_256_GCM_SHA384
> > > Server Temp Key: X25519 253 bits
> > > Application protocol: h2
> > > progress: 10% done
> > > progress: 20% done
> > > progress: 30% done
> > > progress: 40% done
> > > progress: 50% done
> > >
> > > finished in 5.16s, 10.48 req/s, 552B/s
> > > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46
> > > errored, 0 timeout
> > > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx
> > > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings
> > 43.10%),
> > > 0B (0) data
> > >                      min         max         mean         sd        +/-
> > sd
> > > time for request:     1.57ms      9.43ms      2.24ms      1.17ms
> > 94.44%
> > > time for connect:     4.69ms      4.69ms      4.69ms         0us
> >  100.00%
> > > time to 1st byte:        0us         0us         0us         0us
> >  0.00%
> > > req/s           :      10.48       10.48       10.48        0.00
> >  100.00%
> > >
> > > This above configuration always returns 54 succeeded, payload size is
> > 1200B
> > > (1200x54=64800)
> > >
> >
> ------------------------------------------------------------------------------------------------------------------------------
> > > Now reduce the payload and trying the same test,
> > >
> > > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> > > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt
> > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > > starting benchmark...
> > > spawning thread #0: 1 total client(s). 100 total requests
> > > TLS Protocol: TLSv1.3
> > > Cipher: TLS_AES_256_GCM_SHA384
> > > Server Temp Key: X25519 253 bits
> > > Application protocol: h2
> > > progress: 10% done
> > > progress: 20% done
> > > progress: 30% done
> > > progress: 40% done
> > > progress: 50% done
> > > progress: 60% done
> > > progress: 70% done
> > > progress: 80% done
> > >
> > > finished in 5.21s, 16.11 req/s, 839B/s
> > > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16
> > > errored, 0 timeout
> > > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx
> > > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings
> > 43.10%),
> > > 0B (0) data
> > >                      min         max         mean         sd        +/-
> > sd
> > > time for request:     1.43ms      5.80ms      2.04ms       760us
> > 89.29%
> > > time for connect:     5.02ms      5.02ms      5.02ms         0us
> >  100.00%
> > > time to 1st byte:        0us         0us         0us         0us
> >  0.00%
> > > req/s           :      16.11       16.11       16.11        0.00
> >  100.00%
> > >
> > > This above configuration always returns 84 succeeded, payload size is
> > 775B
> > > (775x84=65100)
> > >
> >
> ------------------------------------------------------------------------------------------------------------------------------
> > > Reducing the payload much smaller,
> > >
> > > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d
> > > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt
> > > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > > starting benchmark...
> > > spawning thread #0: 1 total client(s). 200 total requests
> > > TLS Protocol: TLSv1.3
> > > Cipher: TLS_AES_256_GCM_SHA384
> > > Server Temp Key: X25519 253 bits
> > > Application protocol: h2
> > > progress: 10% done
> > > progress: 20% done
> > > progress: 30% done
> > > progress: 40% done
> > > progress: 50% done
> > > progress: 60% done
> > > progress: 70% done
> > > progress: 80% done
> > > progress: 90% done
> > >
> > > finished in 5.41s, 34.40 req/s, 1.73KB/s
> > > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed,
> 14
> > > errored, 0 timeout
> > > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx
> > > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings
> > 43.10%),
> > > 0B (0) data
> > >                      min         max         mean         sd        +/-
> > sd
> > > time for request:     1.18ms     13.49ms      1.91ms      1.13ms
> > 95.16%
> > > time for connect:     5.93ms      5.93ms      5.93ms         0us
> >  100.00%
> > > time to 1st byte:        0us         0us         0us         0us
> >  0.00%
> > > req/s           :      34.41       34.41       34.41        0.00
> >  100.00%
> > >
> > > This above configuration always returns 186 succeeded, payload size is
> > > 356 (356x186=66216)
> > >
> > > On Wed, Mar 6, 2019 at 9:15 PM John Dale <jcdwrrc@gmail.com> wrote:
> > >
> > >> When you run your test(s), does it fail after a certain period of
> > >> time, or just keep on going under a certain number of requests?
> > >>
> > >> Also, to confirm: you're sending 1000 Byte + query strings?
> > >>
> > >> Are you doing anything in the server side component to verify that
> > >> your parameters have been received successfully?
> > >>
> > >> I seems very possible that there is increased overhead parsing the
> > >> request (POST) body.  That's why I was wondering about the dynamics of
> > >> your test case.  If you can achieve a steady load state, either some
> > >> optimization of the POST request parser could be done, or you could
> > >> accept that overhead if it is comparable to other solutions.
> > >>
> > >> On 3/6/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
> > >>> I hope so, I used updated packages/components at the time of
> > development.
> > >>> few may be outdated like tomcat native as I was using 1.2.18 while
> > >>> developing but 1.2.21 got released recently.
> > >>>
> > >>> On Wed, Mar 6, 2019 at 6:18 PM John Dale <jcdwrrc@gmail.com>
wrote:
> > >>>
> > >>>> Have you upgraded to the most recent release of your major version?
> > >>>>
> > >>>> If so, and if this issue still persists, it is something that the
> core
> > >>>> development team might want to look at assuming they can replicate
> the
> > >>>> issue.
> > >>>>
> > >>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
> > >>>>> Sometimes more than 10x
> > >>>>>
> > >>>>> On Tue, Mar 5, 2019 at 10:00 PM John Dale <jcdwrrc@gmail.com>
> wrote:
> > >>>>>
> > >>>>>> How many orders of magnitude slower are the post requests?
> > >>>>>>
> > >>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
> > >>>>>>> I was testing in the localhost
> > >>>>>>>
> > >>>>>>> On Tue, Mar 5, 2019 at 9:32 PM John Dale <jcdwrrc@gmail.com>
> > >> wrote:
> > >>>>>>>
> > >>>>>>>> Are you running your test client (h2load) on the
same machine,
> > >> same
> > >>>>>>>> network, or is it over the net (so, like 20ms latency
on each
> > >>>>>>>> request)?  The reason I ask is that if you are
local
> (especially),
> > >>>>>>>> it
> > >>>>>>>> may queue up too many requests for tomcat to handle
in the
> testing
> > >>>>>>>> period with its thread pool.  Will let you know
if I have any
> > >> other
> > >>>>>>>> ideas.
> > >>>>>>>>
> > >>>>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
> > >>>>>>>>> Bytes
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Mar 5, 2019 at 9:28 PM John Dale <jcdwrrc@gmail.com>
> > >>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> 1000-1500 MB or KB?
> > >>>>>>>>>>
> > >>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
> > >>>>>>>>>>> As per the documentation,
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig
> > >>>>>>>>>>>
> > >>>>>>>>>>> this connector supports maxPostSize,
by default the limit is
> > >>>>>>>>>>> set
> > >>>>>>>>>>> to
> > >>>>>>>> 2MB
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, Mar 5, 2019 at 5:09 AM John
Dale <jcdwrrc@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Does anyone know if this connector
supports maxPostSize
> > >>>>>>>>>>>> parameter?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
> > >>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We have a tomcat instance which
is http2 enabled and it
> > >>>>>>>>>>>>> needs
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>> serve
> > >>>>>>>>>>>>> large number of requests using
multiplexing, so we have
> > >>>>>>>>>>>>> configured
> > >>>>>>>>>>>>> our
> > >>>>>>>>>>>>> instance as follows,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> <Connector port="9191" 
URIEncoding="UTF-8"
> > >>>>>>>>>>>>> sslImplementationName="org.apache.tomcat.util.net
> > >>>>>>>>>>>> .openssl.OpenSSLImplementation"
> > >>>>>>>>>>>>> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> > >>>>>>>>>>>>>                          maxThreads="50000"
> > >>>>>>>>>>>>> SSLEnabled="true"
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
> > >>>>>>>>>>>>>                          compression="on"
> > >>>> minSpareThreads="25"
> > >>>>>>>>>>>>> noCompressionUserAgents="gozilla,
traviata" scheme="https"
> > >>>>>>>>>>>>> secure="true"
> > >>>>>>>>>>>>> keystoreFile="conf/myfile.keystore"
> > >> keystorePass="password"
> > >>>>>>>>>>>>>                          socket.appReadBufSize="81920"
> > >>>>>>>>>>>>> socket.appWriteBufSize="81920"
socket.rxBufSize="251880"
> > >>>>>>>>>>>>> socket.txBufSize="438000">
> > >>>>>>>>>>>>>                         <UpgradeProtocol
compression="on"
> > >>>>>>>>>>>>>
> > >>>>>> maxConcurrentStreamExecution="200"
> > >>>>>>>>>>>>> maxConcurrentStreams="200"
> > >>>>>>>>>>>>> className="org.apache.coyote.http2.Http2Protocol"/>
> > >>>>>>>>>>>>>       </Connector>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> This instance mainly serves
concurrent POST request which
> > >>>> will
> > >>>>>>>>>>>>> have
> > >>>>>>>>>>>> payload
> > >>>>>>>>>>>>> of size, approx 1000-1500,
which can be verified by tomcat
> > >>>>>>>>>>>>> logs
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> org.apache.coyote.http2.Http2Parser.validateFrame
> > >>>>>>>>>>>>> Connection
> > >>>>>> [0],
> > >>>>>>>>>>>>> Stream
> > >>>>>>>>>>>>> [19], Frame type [DATA], Flags
[1], Payload size [*1195*]
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We tested our server with the
help of h2load as follows,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> h2load -n100 -c1 -m100 https://localhost:9191/
-d
> > >>>>>>>>>>>>> '/agentRequest.txt'
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We are getting this error as
follows,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch
> > >>>>>>>>>>>>> Connection
> > >>>>>>>>>>>> [0]
> > >>>>>>>>>>>>>  java.io.IOException: Unable
to unwrap data, invalid
> > >> status
> > >>>>>>>>>>>>> [BUFFER_OVERFLOW]
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> java.base/sun.nio.ch
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> java.base/sun.nio.ch
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> java.base/sun.nio.ch
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .SecureNio2Channel.read(SecureNio2Channel.java:1067)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .SocketWrapperBase.read(SocketWrapperBase.java:1012)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>>>> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .SocketProcessorBase.run(SocketProcessorBase.java:49)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>> org.apache.tomcat.util.net
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> > >>>>>>>>>>>>>         at
> > >>>>>>>>>>>>>
> > >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Why is this error is thrown?
How can I configure tomcat to
> > >>>>>> handle
> > >>>>>>>>>>>>> concurrent POST requests which
have a decent payload?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We have tried with various
java clients like
> > >>>>>>>>>>>>> http-client-5-beta,
> > >>>>>>>>>>>>> jetty
> > >>>>>>>>>>>>> or
> > >>>>>>>>>>>>> okhttp3 and spam requests to
our tomcat using http2
> > >>>>>>>>>>>>> multiplexing
> > >>>>>>>> and
> > >>>>>>>>>> we
> > >>>>>>>>>>>>> found the time taken to process
a requests
> > >>>> increases(sometimes
> > >>>>>>>>>>>>> even
> > >>>>>>>>>>>>> 10x)
> > >>>>>>>>>>>>> when load is increased.
> > >>>>>>>>>>>>> We have tweaked all common
configuration related to http2
> > >>>>>>>>>>>>> on
> > >>>>>> both
> > >>>>>>>>>>>>> client
> > >>>>>>>>>>>>> and server side with no luck.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> But same tomcat configuration
can handle 10s of 1000s of
> > >>>>>>>>>>>>> get
> > >>>>>>>> request
> > >>>>>>>>>>>>> concurrently without a problem,
its only creating problem
> > >>>> with
> > >>>>>>>>>>>>> POST
> > >>>>>>>>>>>>> requests.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> What is wrong in our configuration?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Kindly someone shed some light.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Tomcat - 9.0.16
> > >>>>>>>>>>>>> APR-1.2.18
> > >>>>>>>>>>>>> OpenSSL-1.1.1a
> > >>>>>>>>>>>>> JDK-10.0.2
> > >>>>>>>>>>>>> OS - Ubuntu/Centos
> > >>>>>>>>>>>>> HeapSize - 4GB
> > >>>>>>>>>>>>> RAM -16GB
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Kindly help
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>> *With Regards,*
> > >>>>>>>>>>>>> *Santhosh Kumar J*
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>
> > ---------------------------------------------------------------------
> > >>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > >>>>>>>>>>>> For additional commands, e-mail:
> > >> users-help@tomcat.apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> *With Regards,*
> > >>>>>>>>>>> *Santhosh Kumar J*
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > >>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>> *With Regards,*
> > >>>>>>>>> *Santhosh Kumar J*
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >> ---------------------------------------------------------------------
> > >>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > >>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>> *With Regards,*
> > >>>>>>> *Santhosh Kumar J*
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > ---------------------------------------------------------------------
> > >>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>> --
> > >>>>> *With Regards,*
> > >>>>> *Santhosh Kumar J*
> > >>>>>
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> > >>>>
> > >>>>
> > >>>
> > >>> --
> > >>> *With Regards,*
> > >>> *Santhosh Kumar J*
> > >>>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > >> For additional commands, e-mail: users-help@tomcat.apache.org
> > >>
> > >>
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>


-- 
*With Regards,*
*Santhosh Kumar J*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message