tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rémy Maucherat <r...@apache.org>
Subject Re: Http/2 : Tomcat NIO2 fails on large number of POST request with payload
Date Thu, 07 Mar 2019 16:22:35 GMT
On Thu, Mar 7, 2019 at 2:54 PM Rémy Maucherat <remm@apache.org> wrote:

> On Thu, Mar 7, 2019 at 1:47 PM Mark Thomas <markt@apache.org> wrote:
>
>> On 07/03/2019 07:40, Santhosh Kumar wrote:
>> > From some of the test cases I can safely say that tomcat is hitting some
>> > limits, I have two test cases ran with two diff size of payload and
>> without
>> > any queryParams. The servlet is a empty servlet just returns after
>> > receiving without doing any business side logic
>>
>> Can you repeat those tests with the NIO connector? It would be helpful
>> to know if we should be looking at the HTTP/2 code or the low-level
>> connector I/O code.
>>
>
> I was planning to investigate since I'm hunting NIO2 additional issues
> after the fix for BZ63182. This one looks simpler to reproduce at least
> [assuming there's an issue].
>

Ok, so it's a buffer size issue with vectored IO and SSL, the sizes used
are too optimized.

Rémy


>
> Rémy
>
>
>>
>> Thanks,
>>
>> Mark
>>
>>
>> >
>> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
>> > /home/local/santhosh/A-Test/nghttp2/agentRequest.txt
>> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
>> > starting benchmark...
>> > spawning thread #0: 1 total client(s). 100 total requests
>> > TLS Protocol: TLSv1.3
>> > Cipher: TLS_AES_256_GCM_SHA384
>> > Server Temp Key: X25519 253 bits
>> > Application protocol: h2
>> > progress: 10% done
>> > progress: 20% done
>> > progress: 30% done
>> > progress: 40% done
>> > progress: 50% done
>> >
>> > finished in 5.16s, 10.48 req/s, 552B/s
>> > requests: 100 total, 55 started, 54 done, 54 succeeded, 46 failed, 46
>> > errored, 0 timeout
>> > status codes: 55 2xx, 0 3xx, 0 4xx, 0 5xx
>> > traffic: 2.78KB (2846) total, 1.77KB (1815) headers (space savings
>> 43.10%),
>> > 0B (0) data
>> >                      min         max         mean         sd        +/-
>> sd
>> > time for request:     1.57ms      9.43ms      2.24ms      1.17ms
>> 94.44%
>> > time for connect:     4.69ms      4.69ms      4.69ms         0us
>>  100.00%
>> > time to 1st byte:        0us         0us         0us         0us
>>  0.00%
>> > req/s           :      10.48       10.48       10.48        0.00
>>  100.00%
>> >
>> > This above configuration always returns 54 succeeded, payload size is
>> 1200B
>> > (1200x54=64800)
>> >
>> ------------------------------------------------------------------------------------------------------------------------------
>> > Now reduce the payload and trying the same test,
>> >
>> > h2load -n100 -c1 -m1 --header="Content-Type:application/json" -d
>> > /home/local/santhosh/A-Test/nghttp2/agentRequest2.txt
>> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
>> > starting benchmark...
>> > spawning thread #0: 1 total client(s). 100 total requests
>> > TLS Protocol: TLSv1.3
>> > Cipher: TLS_AES_256_GCM_SHA384
>> > Server Temp Key: X25519 253 bits
>> > Application protocol: h2
>> > progress: 10% done
>> > progress: 20% done
>> > progress: 30% done
>> > progress: 40% done
>> > progress: 50% done
>> > progress: 60% done
>> > progress: 70% done
>> > progress: 80% done
>> >
>> > finished in 5.21s, 16.11 req/s, 839B/s
>> > requests: 100 total, 85 started, 84 done, 84 succeeded, 16 failed, 16
>> > errored, 0 timeout
>> > status codes: 85 2xx, 0 3xx, 0 4xx, 0 5xx
>> > traffic: 4.27KB (4376) total, 2.74KB (2805) headers (space savings
>> 43.10%),
>> > 0B (0) data
>> >                      min         max         mean         sd        +/-
>> sd
>> > time for request:     1.43ms      5.80ms      2.04ms       760us
>> 89.29%
>> > time for connect:     5.02ms      5.02ms      5.02ms         0us
>>  100.00%
>> > time to 1st byte:        0us         0us         0us         0us
>>  0.00%
>> > req/s           :      16.11       16.11       16.11        0.00
>>  100.00%
>> >
>> > This above configuration always returns 84 succeeded, payload size is
>> 775B
>> > (775x84=65100)
>> >
>> ------------------------------------------------------------------------------------------------------------------------------
>> > Reducing the payload much smaller,
>> >
>> > h2load -n200 -c1 -m1 --header="Content-Type:application/json" -d
>> > /home/local/santhosh/A-Test/nghttp2/agentRequest3.txt
>> > https://localhost:9191/HTTP_2_TEST_APP/Http2Servlet
>> > starting benchmark...
>> > spawning thread #0: 1 total client(s). 200 total requests
>> > TLS Protocol: TLSv1.3
>> > Cipher: TLS_AES_256_GCM_SHA384
>> > Server Temp Key: X25519 253 bits
>> > Application protocol: h2
>> > progress: 10% done
>> > progress: 20% done
>> > progress: 30% done
>> > progress: 40% done
>> > progress: 50% done
>> > progress: 60% done
>> > progress: 70% done
>> > progress: 80% done
>> > progress: 90% done
>> >
>> > finished in 5.41s, 34.40 req/s, 1.73KB/s
>> > requests: 200 total, 187 started, 186 done, 186 succeeded, 14 failed, 14
>> > errored, 0 timeout
>> > status codes: 187 2xx, 0 3xx, 0 4xx, 0 5xx
>> > traffic: 9.35KB (9578) total, 6.03KB (6171) headers (space savings
>> 43.10%),
>> > 0B (0) data
>> >                      min         max         mean         sd        +/-
>> sd
>> > time for request:     1.18ms     13.49ms      1.91ms      1.13ms
>> 95.16%
>> > time for connect:     5.93ms      5.93ms      5.93ms         0us
>>  100.00%
>> > time to 1st byte:        0us         0us         0us         0us
>>  0.00%
>> > req/s           :      34.41       34.41       34.41        0.00
>>  100.00%
>> >
>> > This above configuration always returns 186 succeeded, payload size is
>> > 356 (356x186=66216)
>> >
>> > On Wed, Mar 6, 2019 at 9:15 PM John Dale <jcdwrrc@gmail.com> wrote:
>> >
>> >> When you run your test(s), does it fail after a certain period of
>> >> time, or just keep on going under a certain number of requests?
>> >>
>> >> Also, to confirm: you're sending 1000 Byte + query strings?
>> >>
>> >> Are you doing anything in the server side component to verify that
>> >> your parameters have been received successfully?
>> >>
>> >> I seems very possible that there is increased overhead parsing the
>> >> request (POST) body.  That's why I was wondering about the dynamics of
>> >> your test case.  If you can achieve a steady load state, either some
>> >> optimization of the POST request parser could be done, or you could
>> >> accept that overhead if it is comparable to other solutions.
>> >>
>> >> On 3/6/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>> >>> I hope so, I used updated packages/components at the time of
>> development.
>> >>> few may be outdated like tomcat native as I was using 1.2.18 while
>> >>> developing but 1.2.21 got released recently.
>> >>>
>> >>> On Wed, Mar 6, 2019 at 6:18 PM John Dale <jcdwrrc@gmail.com> wrote:
>> >>>
>> >>>> Have you upgraded to the most recent release of your major version?
>> >>>>
>> >>>> If so, and if this issue still persists, it is something that the
>> core
>> >>>> development team might want to look at assuming they can replicate
>> the
>> >>>> issue.
>> >>>>
>> >>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com> wrote:
>> >>>>> Sometimes more than 10x
>> >>>>>
>> >>>>> On Tue, Mar 5, 2019 at 10:00 PM John Dale <jcdwrrc@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> How many orders of magnitude slower are the post requests?
>> >>>>>>
>> >>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>> >>>>>>> I was testing in the localhost
>> >>>>>>>
>> >>>>>>> On Tue, Mar 5, 2019 at 9:32 PM John Dale <jcdwrrc@gmail.com>
>> >> wrote:
>> >>>>>>>
>> >>>>>>>> Are you running your test client (h2load) on the
same machine,
>> >> same
>> >>>>>>>> network, or is it over the net (so, like 20ms latency
on each
>> >>>>>>>> request)?  The reason I ask is that if you are local
>> (especially),
>> >>>>>>>> it
>> >>>>>>>> may queue up too many requests for tomcat to handle
in the
>> testing
>> >>>>>>>> period with its thread pool.  Will let you know
if I have any
>> >> other
>> >>>>>>>> ideas.
>> >>>>>>>>
>> >>>>>>>> On 3/5/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>> >>>>>>>>> Bytes
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Mar 5, 2019 at 9:28 PM John Dale <jcdwrrc@gmail.com>
>> >>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> 1000-1500 MB or KB?
>> >>>>>>>>>>
>> >>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>> >>>>>>>>>>> As per the documentation,
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support_-_SSLHostConfig
>> >>>>>>>>>>>
>> >>>>>>>>>>> this connector supports maxPostSize,
by default the limit is
>> >>>>>>>>>>> set
>> >>>>>>>>>>> to
>> >>>>>>>> 2MB
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Tue, Mar 5, 2019 at 5:09 AM John
Dale <jcdwrrc@gmail.com>
>> >>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Does anyone know if this connector
supports maxPostSize
>> >>>>>>>>>>>> parameter?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On 3/4/19, Santhosh Kumar <santhosh89j@gmail.com>
wrote:
>> >>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> We have a tomcat instance which
is http2 enabled and it
>> >>>>>>>>>>>>> needs
>> >>>>>>>>>>>>> to
>> >>>>>>>>>>>>> serve
>> >>>>>>>>>>>>> large number of requests using
multiplexing, so we have
>> >>>>>>>>>>>>> configured
>> >>>>>>>>>>>>> our
>> >>>>>>>>>>>>> instance as follows,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> <Connector port="9191"  URIEncoding="UTF-8"
>> >>>>>>>>>>>>> sslImplementationName="org.apache.tomcat.util.net
>> >>>>>>>>>>>> .openssl.OpenSSLImplementation"
>> >>>>>>>>>>>>> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>> >>>>>>>>>>>>>                          maxThreads="50000"
>> >>>>>>>>>>>>> SSLEnabled="true"
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml"
>> >>>>>>>>>>>>>                          compression="on"
>> >>>> minSpareThreads="25"
>> >>>>>>>>>>>>> noCompressionUserAgents="gozilla,
traviata" scheme="https"
>> >>>>>>>>>>>>> secure="true"
>> >>>>>>>>>>>>> keystoreFile="conf/myfile.keystore"
>> >> keystorePass="password"
>> >>>>>>>>>>>>>                          socket.appReadBufSize="81920"
>> >>>>>>>>>>>>> socket.appWriteBufSize="81920"
socket.rxBufSize="251880"
>> >>>>>>>>>>>>> socket.txBufSize="438000">
>> >>>>>>>>>>>>>                         <UpgradeProtocol
compression="on"
>> >>>>>>>>>>>>>
>> >>>>>> maxConcurrentStreamExecution="200"
>> >>>>>>>>>>>>> maxConcurrentStreams="200"
>> >>>>>>>>>>>>> className="org.apache.coyote.http2.Http2Protocol"/>
>> >>>>>>>>>>>>>       </Connector>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> This instance mainly serves
concurrent POST request which
>> >>>> will
>> >>>>>>>>>>>>> have
>> >>>>>>>>>>>> payload
>> >>>>>>>>>>>>> of size, approx 1000-1500, which
can be verified by tomcat
>> >>>>>>>>>>>>> logs
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> org.apache.coyote.http2.Http2Parser.validateFrame
>> >>>>>>>>>>>>> Connection
>> >>>>>> [0],
>> >>>>>>>>>>>>> Stream
>> >>>>>>>>>>>>> [19], Frame type [DATA], Flags
[1], Payload size [*1195*]
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> We tested our server with the
help of h2load as follows,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> h2load -n100 -c1 -m100 https://localhost:9191/
-d
>> >>>>>>>>>>>>> '/agentRequest.txt'
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> We are getting this error as
follows,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch
>> >>>>>>>>>>>>> Connection
>> >>>>>>>>>>>> [0]
>> >>>>>>>>>>>>>  java.io.IOException: Unable
to unwrap data, invalid
>> >> status
>> >>>>>>>>>>>>> [BUFFER_OVERFLOW]
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1041)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> java.base/sun.nio.ch
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> .UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:552)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> java.base/sun.nio.ch
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> java.base/sun.nio.ch
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> .AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1027)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .SecureNio2Channel$2.completed(SecureNio2Channel.java:1000)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .SecureNio2Channel.read(SecureNio2Channel.java:1067)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> .Nio2Endpoint$Nio2SocketWrapper$VectoredIOCompletionHandler.completed(Nio2Endpoint.java:1153)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .Nio2Endpoint$Nio2SocketWrapper.read(Nio2Endpoint.java:1026)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .SocketWrapperBase.read(SocketWrapperBase.java:1012)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> org.apache.coyote.http2.Http2AsyncParser.readFrame(Http2AsyncParser.java:61)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>>>> org.apache.coyote.http2.Http2Parser.readFrame(Http2Parser.java:69)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:322)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:37)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1769)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>> .AbstractEndpoint.processSocket(AbstractEndpoint.java:1048)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:116)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>> org.apache.tomcat.util.net
>> >>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> .SecureNio2Channel$HandshakeWriteCompletionHandler.completed(SecureNio2Channel.java:109)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >>>> java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
>> >>>>>>>>>>>>>         at
>> >>>>>>>>>>>>>
>> >> java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Why is this error is thrown?
How can I configure tomcat to
>> >>>>>> handle
>> >>>>>>>>>>>>> concurrent POST requests which
have a decent payload?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> We have tried with various java
clients like
>> >>>>>>>>>>>>> http-client-5-beta,
>> >>>>>>>>>>>>> jetty
>> >>>>>>>>>>>>> or
>> >>>>>>>>>>>>> okhttp3 and spam requests to
our tomcat using http2
>> >>>>>>>>>>>>> multiplexing
>> >>>>>>>> and
>> >>>>>>>>>> we
>> >>>>>>>>>>>>> found the time taken to process
a requests
>> >>>> increases(sometimes
>> >>>>>>>>>>>>> even
>> >>>>>>>>>>>>> 10x)
>> >>>>>>>>>>>>> when load is increased.
>> >>>>>>>>>>>>> We have tweaked all common configuration
related to http2
>> >>>>>>>>>>>>> on
>> >>>>>> both
>> >>>>>>>>>>>>> client
>> >>>>>>>>>>>>> and server side with no luck.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> But same tomcat configuration
can handle 10s of 1000s of
>> >>>>>>>>>>>>> get
>> >>>>>>>> request
>> >>>>>>>>>>>>> concurrently without a problem,
its only creating problem
>> >>>> with
>> >>>>>>>>>>>>> POST
>> >>>>>>>>>>>>> requests.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> What is wrong in our configuration?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Kindly someone shed some light.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Tomcat - 9.0.16
>> >>>>>>>>>>>>> APR-1.2.18
>> >>>>>>>>>>>>> OpenSSL-1.1.1a
>> >>>>>>>>>>>>> JDK-10.0.2
>> >>>>>>>>>>>>> OS - Ubuntu/Centos
>> >>>>>>>>>>>>> HeapSize - 4GB
>> >>>>>>>>>>>>> RAM -16GB
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Kindly help
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> --
>> >>>>>>>>>>>>> *With Regards,*
>> >>>>>>>>>>>>> *Santhosh Kumar J*
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>>>>>>>>>> For additional commands, e-mail:
>> >> users-help@tomcat.apache.org
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> --
>> >>>>>>>>>>> *With Regards,*
>> >>>>>>>>>>> *Santhosh Kumar J*
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>> ---------------------------------------------------------------------
>> >>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>> *With Regards,*
>> >>>>>>>>> *Santhosh Kumar J*
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >> ---------------------------------------------------------------------
>> >>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>> *With Regards,*
>> >>>>>>> *Santhosh Kumar J*
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> ---------------------------------------------------------------------
>> >>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>> --
>> >>>>> *With Regards,*
>> >>>>> *Santhosh Kumar J*
>> >>>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> *With Regards,*
>> >>> *Santhosh Kumar J*
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >>
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message