tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Dale <jcdw...@gmail.com>
Subject Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Date Wed, 06 Mar 2019 15:44:59 GMT
When you run your test(s), does it fail after a certain period of
time, or just keep on going under a certain number of requests?

Also, to confirm: you're sending 1000 Byte + query strings?

Are you doing anything in the server side component to verify that
your parameters have been received successfully?

I seems very possible that there is increased overhead parsing the
request (POST) body.  That's why I was wondering about the dynamics of
your test case.  If you can achieve a steady load state, either some
optimization of the POST request parser could be done, or you could
accept that overhead if it is comparable to other solutions.

On 3/6/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
> I hope so, I used updated packages/components at the time of development.
> few may be outdated like tomcat native as I was using 1.2.18 while
> developing but 1.2.21 got released recently.
>
> On Wed, Mar 6, 2019 at 6:18 PM John Dale <jcdwrrc@gmail.com> wrote:
>
>> Have you upgraded to the most recent release of your major version?
>>
>> If so, and if this issue still persists, it is something that the core
>> development team might want to look at assuming they can replicate the
>> issue.
>>
>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>> > Sometimes more than 10x
>> >
>> > On Tue, Mar 5, 2019 at 10:00 PM John Dale <jcdwrrc@gmail.com> wrote:
>> >
>> >> How many orders of magnitude slower are the post requests?
>> >>
>> >> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>> >> > I was testing in the localhost
>> >> >
>> >> > On Tue, Mar 5, 2019 at 9:32 PM John Dale <jcdwrrc@gmail.com>
wrote:
>> >> >
>> >> >> Are you running your test client (h2load) on the same machine,
same
>> >> >> network, or is it over the net (so, like 20ms latency on each
>> >> >> request)?  The reason I ask is that if you are local (especially),
>> >> >> it
>> >> >> may queue up too many requests for tomcat to handle in the testing
>> >> >> period with its thread pool.  Will let you know if I have any other
>> >> >> ideas.
>> >> >>
>> >> >> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>> >> >> > Bytes
>> >> >> >
>> >> >> > On Tue, Mar 5, 2019 at 9:28 PM John Dale <jcdwrrc@gmail.com>
>> wrote:
>> >> >> >
>> >> >> >> 1000-1500 MB or KB?
>> >> >> >>
>> >> >> >> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>> >> >> >> > As per the documentation,
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >>
>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig
>> >> >> >> >
>> >> >> >> > this connector supports maxPostSize, by default the
limit is
>> >> >> >> > set
>> >> >> >> > to
>> >> >> 2MB
>> >> >> >> >
>> >> >> >> > On Tue, Mar 5, 2019 at 5:09 AM John Dale <jcdwrrc@gmail.com>
>> >> wrote:
>> >> >> >> >
>> >> >> >> >> Does anyone know if this connector supports maxPostSize
>> >> >> >> >> parameter?
>> >> >> >> >>
>> >> >> >> >> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>> >> >> >> >> > Hi,
>> >> >> >> >> >
>> >> >> >> >> > We have a tomcat instance which is http2
enabled and it
>> >> >> >> >> > needs
>> >> >> >> >> > to
>> >> >> >> >> > serve
>> >> >> >> >> > large number of requests using multiplexing,
so we have
>> >> >> >> >> > configured
>> >> >> >> >> > our
>> >> >> >> >> > instance as follows,
>> >> >> >> >> >
>> >> >> >> >> > <Connector port="9191"  URIEncoding="UTF-8"
>> >> >> >> >> > sslImplementationName="org.apache.tomcat.util.net
>> >> >> >> >> .openssl.OpenSSLImplementation"
>> >> >> >> >> > protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>> >> >> >> >> >                          maxThreads="50000"
>> >> >> >> >> > SSLEnabled="true"
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
>> >> >> >> >> >                          compression="on"
>> minSpareThreads="25"
>> >> >> >> >> > noCompressionUserAgents="gozilla, traviata"
scheme="https"
>> >> >> >> >> > secure="true"
>> >> >> >> >> > keystoreFile="conf/myfile.keystore" keystorePass="password"
>> >> >> >> >> >                          socket.appReadBufSize="81920"
>> >> >> >> >> > socket.appWriteBufSize="81920" socket.rxBufSize="251880"
>> >> >> >> >> > socket.txBufSize="438000">
>> >> >> >> >> >                         <UpgradeProtocol
compression="on"
>> >> >> >> >> >
>> >> maxConcurrentStreamExecution="200"
>> >> >> >> >> > maxConcurrentStreams="200"
>> >> >> >> >> > className="org.apache.coyote.http2.Http2Protocol"/>
>> >> >> >> >> >       </Connector>
>> >> >> >> >> >
>> >> >> >> >> > This instance mainly serves concurrent POST
request which
>> will
>> >> >> >> >> > have
>> >> >> >> >> payload
>> >> >> >> >> > of size, approx 1000-1500, which can be
verified by tomcat
>> >> >> >> >> > logs
>> >> >> >> >> >
>> >> >> >> >> > org.apache.coyote.http2.Http2Parser.validateFrame
>> >> >> >> >> > Connection
>> >> [0],
>> >> >> >> >> > Stream
>> >> >> >> >> > [19], Frame type [DATA], Flags [1], Payload
size [*1195*]
>> >> >> >> >> >
>> >> >> >> >> > We tested our server with the help of h2load
as follows,
>> >> >> >> >> >
>> >> >> >> >> > h2load -n100 -c1 -m100 https://localhost:9191/
-d
>> >> >> >> >> > '/agentRequest.txt'
>> >> >> >> >> >
>> >> >> >> >> > We are getting this error as follows,
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch
>> >> >> >> >> > Connection
>> >> >> >> >> [0]
>> >> >> >> >> >  java.io.IOException: Unable to unwrap data,
invalid status
>> >> >> >> >> > [BUFFER_OVERFLOW]
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>> >> >> >> >> >         at
>> >> >> >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>> >> >> >> >> >         at
>> >> >> >> >> > java.base/sun.nio.ch
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
>> >> >> >> >> >         at
>> >> >> >> >> > java.base/sun.nio.ch
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>> >> >> >> >> >         at
>> >> >> >> >> > java.base/sun.nio.ch
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .SecureNio2Channel.read(SecureNio2Channel.java:1067)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .SocketWrapperBase.read(SocketWrapperBase.java:1012)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> >> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
>> >> >> >> >> >         at
>> >> >> >> >> > org.apache.tomcat.util.net
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >>
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
>> >> >> >> >> >         at
>> >> >> >> >> >
>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>> >> >> >> >> >         at
>> >> >> >> >> > java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>> >> >> >> >> >
>> >> >> >> >> > Why is this error is thrown? How can I configure
tomcat to
>> >> handle
>> >> >> >> >> > concurrent POST requests which have a decent
payload?
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > We have tried with various java clients
like
>> >> >> >> >> > http-client-5-beta,
>> >> >> >> >> > jetty
>> >> >> >> >> > or
>> >> >> >> >> > okhttp3 and spam requests to our tomcat
using http2
>> >> >> >> >> > multiplexing
>> >> >> and
>> >> >> >> we
>> >> >> >> >> > found the time taken to process a requests
>> increases(sometimes
>> >> >> >> >> > even
>> >> >> >> >> > 10x)
>> >> >> >> >> > when load is increased.
>> >> >> >> >> > We have tweaked all common configuration
related to http2
>> >> >> >> >> > on
>> >> both
>> >> >> >> >> > client
>> >> >> >> >> > and server side with no luck.
>> >> >> >> >> >
>> >> >> >> >> > But same tomcat configuration can handle
10s of 1000s of
>> >> >> >> >> > get
>> >> >> request
>> >> >> >> >> > concurrently without a problem, its only
creating problem
>> with
>> >> >> >> >> > POST
>> >> >> >> >> > requests.
>> >> >> >> >> >
>> >> >> >> >> > What is wrong in our configuration?
>> >> >> >> >> >
>> >> >> >> >> > Kindly someone shed some light.
>> >> >> >> >> >
>> >> >> >> >> > Tomcat - 9.0.16
>> >> >> >> >> > APR-1.2.18
>> >> >> >> >> > OpenSSL-1.1.1a
>> >> >> >> >> > JDK-10.0.2
>> >> >> >> >> > OS - Ubuntu/Centos
>> >> >> >> >> > HeapSize - 4GB
>> >> >> >> >> > RAM -16GB
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > Kindly help
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > *With Regards,*
>> >> >> >> >> > *Santhosh Kumar J*
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >> >>
>> >> ---------------------------------------------------------------------
>> >> >> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> >> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > *With Regards,*
>> >> >> >> > *Santhosh Kumar J*
>> >> >> >> >
>> >> >> >>
>> >> >> >>
>> ---------------------------------------------------------------------
>> >> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >> >>
>> >> >> >> --
>> >> >> > *With Regards,*
>> >> >> > *Santhosh Kumar J*
>> >> >> >
>> >> >>
>> >> >> ---------------------------------------------------------------------
>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >>
>> >> >> --
>> >> > *With Regards,*
>> >> > *Santhosh Kumar J*
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >>
>> >
>> > --
>> > *With Regards,*
>> > *Santhosh Kumar J*
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> --
> *With Regards,*
> *Santhosh Kumar J*
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message