tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rémy Maucherat <r...@apache.org>
Subject Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Date Thu, 07 Mar 2019 13:54:22 GMT
On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas <markt@apache.org> wrote:

> On 07/03/2019 07:40, Santhosh Kumar wrote:
> > From some of the test cases I can safely say that tomcat is hitting some
> > limits, I have two test cases ran with two diff size of payload and
> without
> > any queryParams. The servlet is a empty servlet just returns after
> > receiving without doing any business side logic
>
> Can you repeat those tests with the NIO connector? It would be helpful
> to know if we should be looking at the HTTP/2 code or the low-level
> connector I/O code.
>

I was planning to investigate since I'm hunting NIO2 additional issues
after the fix for BZ63182. This one looks simpler to reproduce at least
[assuming there's an issue].

Rémy


>
> Thanks,
>
> Mark
>
>
> >
> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt
> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > starting benchmark...
> > spawning thread #0: 1 total client(s). 100 total requests
> > TLS Protocol: TLSv1.3
> > Cipher: TLS_AES_256_GCM_SHA384
> > Server Temp Key: X25519 253 bits
> > Application protocol: h2
> > progress: 10% done
> > progress: 20% done
> > progress: 30% done
> > progress: 40% done
> > progress: 50% done
> >
> > finished in 5.16s, 10.48 req/s, 552B/s
> > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46
> > errored, 0 timeout
> > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx
> > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings
> 43.10%),
> > 0B (0) data
> >                      min         max         mean         sd        +/-
> sd
> > time for request:     1.57ms      9.43ms      2.24ms      1.17ms
> 94.44%
> > time for connect:     4.69ms      4.69ms      4.69ms         0us
>  100.00%
> > time to 1st byte:        0us         0us         0us         0us
>  0.00%
> > req/s           :      10.48       10.48       10.48        0.00
>  100.00%
> >
> > This above configuration always returns 54 succeeded, payload size is
> 1200B
> > (1200x54=64800)
> >
> ------------------------------------------------------------------------------------------------------------------------------
> > Now reduce the payload and trying the same test,
> >
> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
> > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt
> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > starting benchmark...
> > spawning thread #0: 1 total client(s). 100 total requests
> > TLS Protocol: TLSv1.3
> > Cipher: TLS_AES_256_GCM_SHA384
> > Server Temp Key: X25519 253 bits
> > Application protocol: h2
> > progress: 10% done
> > progress: 20% done
> > progress: 30% done
> > progress: 40% done
> > progress: 50% done
> > progress: 60% done
> > progress: 70% done
> > progress: 80% done
> >
> > finished in 5.21s, 16.11 req/s, 839B/s
> > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16
> > errored, 0 timeout
> > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx
> > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings
> 43.10%),
> > 0B (0) data
> >                      min         max         mean         sd        +/-
> sd
> > time for request:     1.43ms      5.80ms      2.04ms       760us
> 89.29%
> > time for connect:     5.02ms      5.02ms      5.02ms         0us
>  100.00%
> > time to 1st byte:        0us         0us         0us         0us
>  0.00%
> > req/s           :      16.11       16.11       16.11        0.00
>  100.00%
> >
> > This above configuration always returns 84 succeeded, payload size is
> 775B
> > (775x84=65100)
> >
> ------------------------------------------------------------------------------------------------------------------------------
> > Reducing the payload much smaller,
> >
> > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d
> > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt
> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
> > starting benchmark...
> > spawning thread #0: 1 total client(s). 200 total requests
> > TLS Protocol: TLSv1.3
> > Cipher: TLS_AES_256_GCM_SHA384
> > Server Temp Key: X25519 253 bits
> > Application protocol: h2
> > progress: 10% done
> > progress: 20% done
> > progress: 30% done
> > progress: 40% done
> > progress: 50% done
> > progress: 60% done
> > progress: 70% done
> > progress: 80% done
> > progress: 90% done
> >
> > finished in 5.41s, 34.40 req/s, 1.73KB/s
> > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14
> > errored, 0 timeout
> > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx
> > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings
> 43.10%),
> > 0B (0) data
> >                      min         max         mean         sd        +/-
> sd
> > time for request:     1.18ms     13.49ms      1.91ms      1.13ms
> 95.16%
> > time for connect:     5.93ms      5.93ms      5.93ms         0us
>  100.00%
> > time to 1st byte:        0us         0us         0us         0us
>  0.00%
> > req/s           :      34.41       34.41       34.41        0.00
>  100.00%
> >
> > This above configuration always returns 186 succeeded, payload size is
> > 356 (356x186=66216)
> >
> > On Wed, Mar 6, 2019 at 9:15 PM John Dale <jcdwrrc@gmail.com> wrote:
> >
> >> When you run your test(s), does it fail after a certain period of
> >> time, or just keep on going under a certain number of requests?
> >>
> >> Also, to confirm: you're sending 1000 Byte + query strings?
> >>
> >> Are you doing anything in the server side component to verify that
> >> your parameters have been received successfully?
> >>
> >> I seems very possible that there is increased overhead parsing the
> >> request (POST) body.  That's why I was wondering about the dynamics of
> >> your test case.  If you can achieve a steady load state, either some
> >> optimization of the POST request parser could be done, or you could
> >> accept that overhead if it is comparable to other solutions.
> >>
> >> On 3/6/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
> >>> I hope so, I used updated packages/components at the time of
> development.
> >>> few may be outdated like tomcat native as I was using 1.2.18 while
> >>> developing but 1.2.21 got released recently.
> >>>
> >>> On Wed, Mar 6, 2019 at 6:18 PM John Dale <jcdwrrc@gmail.com> wrote:
> >>>
> >>>> Have you upgraded to the most recent release of your major version?
> >>>>
> >>>> If so, and if this issue still persists, it is something that the core
> >>>> development team might want to look at assuming they can replicate the
> >>>> issue.
> >>>>
> >>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
> >>>>> Sometimes more than 10x
> >>>>>
> >>>>> On Tue, Mar 5, 2019 at 10:00 PM John Dale <jcdwrrc@gmail.com>
wrote:
> >>>>>
> >>>>>> How many orders of magnitude slower are the post requests?
> >>>>>>
> >>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
> >>>>>>> I was testing in the localhost
> >>>>>>>
> >>>>>>> On Tue, Mar 5, 2019 at 9:32 PM John Dale <jcdwrrc@gmail.com>
> >> wrote:
> >>>>>>>
> >>>>>>>> Are you running your test client (h2load) on the same
machine,
> >> same
> >>>>>>>> network, or is it over the net (so, like 20ms latency
on each
> >>>>>>>> request)?  The reason I ask is that if you are local
(especially),
> >>>>>>>> it
> >>>>>>>> may queue up too many requests for tomcat to handle
in the testing
> >>>>>>>> period with its thread pool.  Will let you know if I
have any
> >> other
> >>>>>>>> ideas.
> >>>>>>>>
> >>>>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
> >>>>>>>>> Bytes
> >>>>>>>>>
> >>>>>>>>> On Tue, Mar 5, 2019 at 9:28 PM John Dale <jcdwrrc@gmail.com>
> >>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> 1000-1500 MB or KB?
> >>>>>>>>>>
> >>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
> >>>>>>>>>>> As per the documentation,
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig
> >>>>>>>>>>>
> >>>>>>>>>>> this connector supports maxPostSize, by
default the limit is
> >>>>>>>>>>> set
> >>>>>>>>>>> to
> >>>>>>>> 2MB
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Mar 5, 2019 at 5:09 AM John Dale
<jcdwrrc@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Does anyone know if this connector supports
maxPostSize
> >>>>>>>>>>>> parameter?
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We have a tomcat instance which
is http2 enabled and it
> >>>>>>>>>>>>> needs
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>> serve
> >>>>>>>>>>>>> large number of requests using multiplexing,
so we have
> >>>>>>>>>>>>> configured
> >>>>>>>>>>>>> our
> >>>>>>>>>>>>> instance as follows,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <Connector port="9191"  URIEncoding="UTF-8"
> >>>>>>>>>>>>> sslImplementationName="org.apache.tomcat.util.net
> >>>>>>>>>>>> .openssl.OpenSSLImplementation"
> >>>>>>>>>>>>> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> >>>>>>>>>>>>>                          maxThreads="50000"
> >>>>>>>>>>>>> SSLEnabled="true"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
> >>>>>>>>>>>>>                          compression="on"
> >>>> minSpareThreads="25"
> >>>>>>>>>>>>> noCompressionUserAgents="gozilla,
traviata" scheme="https"
> >>>>>>>>>>>>> secure="true"
> >>>>>>>>>>>>> keystoreFile="conf/myfile.keystore"
> >> keystorePass="password"
> >>>>>>>>>>>>>                          socket.appReadBufSize="81920"
> >>>>>>>>>>>>> socket.appWriteBufSize="81920" socket.rxBufSize="251880"
> >>>>>>>>>>>>> socket.txBufSize="438000">
> >>>>>>>>>>>>>                         <UpgradeProtocol
compression="on"
> >>>>>>>>>>>>>
> >>>>>> maxConcurrentStreamExecution="200"
> >>>>>>>>>>>>> maxConcurrentStreams="200"
> >>>>>>>>>>>>> className="org.apache.coyote.http2.Http2Protocol"/>
> >>>>>>>>>>>>>       </Connector>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This instance mainly serves concurrent
POST request which
> >>>> will
> >>>>>>>>>>>>> have
> >>>>>>>>>>>> payload
> >>>>>>>>>>>>> of size, approx 1000-1500, which
can be verified by tomcat
> >>>>>>>>>>>>> logs
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> org.apache.coyote.http2.Http2Parser.validateFrame
> >>>>>>>>>>>>> Connection
> >>>>>> [0],
> >>>>>>>>>>>>> Stream
> >>>>>>>>>>>>> [19], Frame type [DATA], Flags [1],
Payload size [*1195*]
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We tested our server with the help
of h2load as follows,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> h2load -n100 -c1 -m100 https://localhost:9191/
-d
> >>>>>>>>>>>>> '/agentRequest.txt'
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We are getting this error as follows,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch
> >>>>>>>>>>>>> Connection
> >>>>>>>>>>>> [0]
> >>>>>>>>>>>>>  java.io.IOException: Unable to
unwrap data, invalid
> >> status
> >>>>>>>>>>>>> [BUFFER_OVERFLOW]
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> java.base/sun.nio.ch
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> java.base/sun.nio.ch
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> java.base/sun.nio.ch
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .SecureNio2Channel.read(SecureNio2Channel.java:1067)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .SocketWrapperBase.read(SocketWrapperBase.java:1012)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>>>> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .SocketProcessorBase.run(SocketProcessorBase.java:49)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>> org.apache.tomcat.util.net
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> >>>>>>>>>>>>>         at
> >>>>>>>>>>>>>
> >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Why is this error is thrown? How
can I configure tomcat to
> >>>>>> handle
> >>>>>>>>>>>>> concurrent POST requests which have
a decent payload?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We have tried with various java
clients like
> >>>>>>>>>>>>> http-client-5-beta,
> >>>>>>>>>>>>> jetty
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>> okhttp3 and spam requests to our
tomcat using http2
> >>>>>>>>>>>>> multiplexing
> >>>>>>>> and
> >>>>>>>>>> we
> >>>>>>>>>>>>> found the time taken to process
a requests
> >>>> increases(sometimes
> >>>>>>>>>>>>> even
> >>>>>>>>>>>>> 10x)
> >>>>>>>>>>>>> when load is increased.
> >>>>>>>>>>>>> We have tweaked all common configuration
related to http2
> >>>>>>>>>>>>> on
> >>>>>> both
> >>>>>>>>>>>>> client
> >>>>>>>>>>>>> and server side with no luck.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> But same tomcat configuration can
handle 10s of 1000s of
> >>>>>>>>>>>>> get
> >>>>>>>> request
> >>>>>>>>>>>>> concurrently without a problem,
its only creating problem
> >>>> with
> >>>>>>>>>>>>> POST
> >>>>>>>>>>>>> requests.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> What is wrong in our configuration?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Kindly someone shed some light.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Tomcat - 9.0.16
> >>>>>>>>>>>>> APR-1.2.18
> >>>>>>>>>>>>> OpenSSL-1.1.1a
> >>>>>>>>>>>>> JDK-10.0.2
> >>>>>>>>>>>>> OS - Ubuntu/Centos
> >>>>>>>>>>>>> HeapSize - 4GB
> >>>>>>>>>>>>> RAM -16GB
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Kindly help
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> *With Regards,*
> >>>>>>>>>>>>> *Santhosh Kumar J*
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>>>>>>> For additional commands, e-mail:
> >> users-help@tomcat.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> *With Regards,*
> >>>>>>>>>>> *Santhosh Kumar J*
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>> *With Regards,*
> >>>>>>>>> *Santhosh Kumar J*
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>>>
> >>>>>>>> --
> >>>>>>> *With Regards,*
> >>>>>>> *Santhosh Kumar J*
> >>>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> --
> >>>>> *With Regards,*
> >>>>> *Santhosh Kumar J*
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>>
> >>>
> >>> --
> >>> *With Regards,*
> >>> *Santhosh Kumar J*
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message