tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier (tomcat) ...@ice-sa.com>
Subject Re: Strange URL rewrite when reverse proxy with Apache HTTP Server
Date Thu, 23 Feb 2017 23:06:14 GMT
On 23.02.2017 21:17, Aaron Gray wrote:
> Another weird thing...
>
> If you go to:
> https://loadbalancer.domain.com/SelfService
> If I am viewing the headers immedaitely I see 302 not found.

302 is not "not found". It is "Found", and it is a Redirect response.
See : https://en.wikipedia.org/wiki/HTTP_302
for a good explanation.
(But stop after the second paragraph, otherwise you'll get confused again..)

  Then I wait
> the 30 sec for it to totally fail and change the URL in the browser to
> https://loadbalancer.domain.com:232700/SelfService
>
> If you just remove the :232700 and hit Enter, it then loads properly.  See
> immedaitly 200 OK and it is fine.  Don't navigate anywhere else like to
> https://loadbalancer.domain.com/static or else you'll have to repeat the
> same thing as above with the :232700 and removing it, hit Enter.
>
> Weird.
>
> Of course none of this is an issue if you dont hit it with the F5 Load
> Balancer to start.

Yes. There seems to be something weird *when a Redirect response is being sent back by 
tomcat*, and has to go back through the load balancer.
And it may be linked to the fact that the load-balancer does not seem to remove this 
:23270 port which eventually reaches the browser.



>
> On Thu, Feb 23, 2017 at 10:24 AM, Aaron Gray <aaronmgray@gmail.com> wrote:
>
>> I randomly tried https://loadbalancer.domain.com/SelfService/index.html
>>
>> It works perfectly.  No issues.  I have no idea why its needed to add on
>> the /index.html (i'm lucky this exists in the tomcat .war, as it was a
>> guess).  This is visually not appealing, but workable.  Maybe I can do some
>> url rewrite or something, IF this cant be fixed normally?
>>
>> On Thu, Feb 23, 2017 at 10:10 AM, Aaron Gray <aaronmgray@gmail.com> wrote:
>>
>>> Okay, guys, please accept my most sincere apology on this thing.  I just
>>> figured 1 thing out.
>>>
>>> browser -> f5:443 -> httpd (23270, https) -> tomcat (http)
>>> using the /static works! (see below)
>>>
>>> If /static/index.html wasnt specified, then it hangs, then comes back as
>>> https://loadbalancer.domain.com:23270/static
>>> But if you go https://loadbalancer.domain.com/static/index.html it
>>> works!!!
>>>
>>> DirectoryIndex index.html
>>> is present.  because if you just go to https://webserver1:23270/static
>>> it works fine, no need for the extra /index.html
>>>
>>> not sure why that is different there.
>>>
>>> I only found this out using curl and just happened to think of it.
>>> So you see the difference, from the tomcat access log, in the two
>>> variations.
>>>
>>> 172.18.160.66 - - [23/Feb/2017:10:00:48 -0800] "GET /static HTTP/1.1" 302
>>> -
>>> 172.18.160.66 - - [23/Feb/2017:10:03:04 -0800] "GET /static/index.html
>>> HTTP/1.1" 200 32
>>>
>>> Still /SelfService wont work though
>>> I cant think of a way to do /SelfService/something...
>>> Since it is a web app and i am no developer and its all from the vendor
>>> (BMC Software) and it never shows naything other than /SelfService in the
>>> URL.  I dont think its a frame, but once again i dont know these things.  I
>>> know they told me it is HTML5, as it was using Silverlight back in the
>>> older release.
>>>
>>>
>>> On Thu, Feb 23, 2017 at 9:22 AM, Aaron Gray <aaronmgray@gmail.com> wrote:
>>>
>>>> Working with my F5 guy, we had an idea, since 80/tcp and 443/tcp were
>>>> already open to the VIP on the F5, we simply turned of 80 -> 443 redirect
>>>> on the F5, and then configured the F5 to use the non-HTTP port in Apache
>>>> HTTP Server (the two backend servers).  So its HTTP the entire way through,
>>>> and the same issue persisted.
>>>>
>>>> browser -> F5(http) -> httpd(http port) -> tomcat(http port, nonssl)
>>>>
>>>> So i think i can rule out any possibility that the fact I was
>>>> terminating SSL in Apache HTTP Server then proxying clear-text back as a
>>>> reason.  It worked fine like I said if you just go direct to the HTTP
>>>> Server and slip the F5.   And F5 works landing the /healthcheck.html (HTTP
>>>> KeepAlive in the root, which isnt proxied), so all good there.  Just cant
>>>> /SelfService (configured w/ Proxy)
>>>>
>>>> thanks for your support here though.
>>>>
>>>> On Thu, Feb 23, 2017 at 9:12 AM, Aaron Gray <aaronmgray@gmail.com>
>>>> wrote:
>>>>
>>>>> SSLProxyEngine On
>>>>> Was already turned on this entire time inside the ssl.conf (I include
>>>>> it) VirtualHost section.
>>>>>
>>>>> I am debating turning on HTTPS in Tomcat on the backend 10.x.x.x app
>>>>> server, and then HTTPS the whole way through and see if that makes any
>>>>> difference.  I may need to request a new firewall to be opened, which
is
>>>>> not able to open until 3/2.  Gonna see.
>>>>>
>>>>>
>>>>> On Thu, Feb 23, 2017 at 8:22 AM, Antonio S. Cofiño <cofinoa@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> On 23/02/17 12:43, André Warnier (tomcat) wrote:
>>>>>>
>>>>>>> On 22.02.2017 19:22, Aaron Gray wrote:
>>>>>>>
>>>>>>>> So this is interesting.
>>>>>>>>
>>>>>>>> So from HTTP server #1 (172.1.1.1 example) I hit:
>>>>>>>> https://172.1.1.1:23270/static and I see this in the HTTP
log:
>>>>>>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/
HTTP/1.1"
>>>>>>>> 200 32
>>>>>>>> I see this in the Tomcat log:
>>>>>>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/
HTTP/1.1"
>>>>>>>> 200 32
>>>>>>>> ((( Yes I know they look identical, but the logformat is
the same,
>>>>>>>> so it
>>>>>>>> comes out looking the same, but they are  two diff log files
totally
>>>>>>>> )))
>>>>>>>>
>>>>>>>> I then hit the https://loadbalancer.domain.com/static from
my Win10
>>>>>>>> laptop
>>>>>>>>    (10.1.1.2 example)
>>>>>>>> and I see this in the HTTP server #1 log, but NOTHING in
the tomcat
>>>>>>>> access
>>>>>>>> log.
>>>>>>>> 10.1.1.2 - - [22/Feb/2017:10:20:02 -0800] "-" 408 -
>>>>>>>>
>>>>>>>> So 408.  Timeout.  Hmm... Why?  We KNOW that it can connect
from
>>>>>>>> http ->
>>>>>>>> tomcat:18080 perfectly.   So is it a timeout BACK to the
F5? That i
>>>>>>>> dont
>>>>>>>> know yet.
>>>>>>>>
>>>>>>>
>>>>>>> It looks as if, in that case, httpd is trying to connect to tomcat,
>>>>>>> but
>>>>>>> - either it /is/ connecting, and sending the request, but tomcat
>>>>>>> never answers
>>>>>>> - or it is connecting to something else, which isn't tomcat
>>>>>>> (but which accepts the request and doesn't answer either)
>>>>>>>
>>>>>>> I am not *sure*, but I believe, that if httpd was trying to connect
>>>>>>> to something that isn't tomcat, and is not listening on port
18080, it
>>>>>>> would get a different error, and a different logfile message.
 So it would
>>>>>>> seem that at least the TCP connection works.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Is the F5 loadbalancer configured to make reverse proxy to a SSL
http?
>>>>>> The directive SSLProxyEngine onthe F5 loadbalencer has to be enabled.
>>>>>> http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxyengine
>>>>>>
>>>>>> If the loadbalancer it's delegating the SSL work to the server1
>>>>>> server1 https listen = 172.1.1.1:23270
>>>>>>
>>>>>> then the F5 could be connecting as http regular TCP connection to
a
>>>>>> SSL TCP one and therefor the SSL handshake it's not happening.
>>>>>>
>>>>>> Antonio
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Now there is still a puzzle there : when you connect as
>>>>>>> browser --> httpd --> tomcat
>>>>>>> then everything works : httpd connects to tomcat, sends a request,
>>>>>>> gets a response, and returns it to the browser.
>>>>>>>
>>>>>>> But when you do the same as :
>>>>>>> browser --> F5 --> httpd --> tomcat
>>>>>>> it "does not work", and the problem seems to be at httpd -->
tomat
>>>>>>> Why would it make a difference in httpd --> tomcat, if it
is either
>>>>>>> the browser or the F5 which sends the request to httpd ?
>>>>>>> the httpd --> tomcat connection should be the same, no matter
what.
>>>>>>>
>>>>>>> Hmm, for now as puzzled as you are.
>>>>>>> Maybe we have a quantum phenomenon here, where the mere fact
of you
>>>>>>> looking at the logfile, changes what happened beforehand. Just
kidding.
>>>>>>>
>>>>>>> Informational note : the Apache access log line, is written when
>>>>>>> Apache *has finished* processing the request (including the possible
>>>>>>> round-trip through tomcat) and has sent the response to the client.
(That's
>>>>>>> because, for example, it can also write the time spent, and the
size of the
>>>>>>> response; so it has to know those before it can write the log
line).
>>>>>>> I presume that it is the same for tomcat.
>>>>>>> Just mentioning this, so that you would maybe have a second look
at
>>>>>>> the tomcat access log, after a few minutes, to make sure that
there really
>>>>>>> is no corresponding line there.
>>>>>>> It could be that tomcat actually /is/ receiving the request,
and
>>>>>>> processing it, but that it takes a long time, and that httpd
tires of
>>>>>>> waiting and already times out in the meantime. In such a case,
you would
>>>>>>> probably see an error some time later in the tomcat error log,
when it is
>>>>>>> trying to eventually return the response, and finds a closed
connection
>>>>>>> instead.
>>>>>>> Now again, why it would do that in one case and not the other,
is
>>>>>>> still a mystery.
>>>>>>>
>>>>>>> Another note : in the "timeout" log line above, httpd is not
showing
>>>>>>> the request URL which triggered this timeout.
>>>>>>> a) it may be showing some additional information in the Apache
error
>>>>>>> log
>>>>>>> b) these exists an httpd add-on module (mod_log_forensic ?) which
>>>>>>> could provide more details in the log, about what happens precisely
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Feb 21, 2017 at 3:39 PM, André Warnier (tomcat)
<
>>>>>>>> aw@ice-sa.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On 21.02.2017 23:28, Aaron Gray wrote:
>>>>>>>>>
>>>>>>>>> Antonio:  The Tomcat server has no knowledge of the F5,
or that it
>>>>>>>>>> is
>>>>>>>>>> being
>>>>>>>>>> fronted by an Apache HTTP Server.  I do SSL termination
in Apache
>>>>>>>>>> HTTP
>>>>>>>>>> Server, and clear-text from HTTP to Tomcat.
>>>>>>>>>> My redirect port for the normal HTTP listen in Tomcat
is commented
>>>>>>>>>> out.
>>>>>>>>>>        <Connector port="18080" protocol="HTTP/1.1"
>>>>>>>>>>                   connectionTimeout="20000" />
>>>>>>>>>>        <!-- A "Connector" using the shared thread
pool-->
>>>>>>>>>>        <!--
>>>>>>>>>>        <Connector executor="tomcatThreadPool"
>>>>>>>>>>                   port="8080" protocol="HTTP/1.1"
>>>>>>>>>>                   connectionTimeout="20000"
>>>>>>>>>>                   redirectPort="8443" />
>>>>>>>>>>        -->
>>>>>>>>>>
>>>>>>>>>> Andre:
>>>>>>>>>> The URL I am using is https://loadbalancer.domain.com
>>>>>>>>>> It is listening on port 80 and 443, if you hit 80,
internally it
>>>>>>>>>> redirects
>>>>>>>>>> you to 443.  No SSL cert on the F5 load balancer.
 It simply sends
>>>>>>>>>> the
>>>>>>>>>> traffic to one of the two HTTP servers (round-robin,
also tried
>>>>>>>>>> persistence, no difference).  The HTTP server is
listening only
>>>>>>>>>> for HTTPS
>>>>>>>>>> on 23270/tcp.
>>>>>>>>>> When I access these DMZ webservers via the F5 load
balancer (to
>>>>>>>>>> which I
>>>>>>>>>> dont have access to, but the network folks configure
for me), it
>>>>>>>>>> hangs.
>>>>>>>>>> Eventually returns: server1 https listen = 172.1.1.1:23270
>>>>>>>>>>
>>>>>>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>>>>>>> cant load.
>>>>>>>>>>
>>>>>>>>>> No idea why the URL is being re-written with the
":23270".
>>>>>>>>>> I added static content to the server.xml on 10.1.1.1
(Tomcat) to
>>>>>>>>>> test:
>>>>>>>>>> <Context docBase="/path/to/tomcat/static" path="/static"
/>
>>>>>>>>>> Then put a simple index.html in there.  Accessing
via the Apache
>>>>>>>>>> Web
>>>>>>>>>> Servers works fine, but if you hit it with the Load
Balancer it
>>>>>>>>>> once
>>>>>>>>>> again
>>>>>>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>>>>>>
>>>>>>>>>> Hitting https://loadbalancer.domain.com
>>>>>>>>>> I see my "Hello world!" which is all that is in index.html.
This
>>>>>>>>>> is the
>>>>>>>>>> DocumentRoot of HTTP, and *not* proxied over at this
time.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> So in this case, there is no delay, and you get the Apache
>>>>>>>>> httpd-hosted
>>>>>>>>> "index.html" containing "Hello World. Right ?
>>>>>>>>>
>>>>>>>>>     Only
>>>>>>>>>
>>>>>>>>> /SelfService and /static are proxied
>>>>>>>>>> /static just being my test of static content, but
still served up
>>>>>>>>>> by
>>>>>>>>>> Tomcat..
>>>>>>>>>>
>>>>>>>>>> It's exactly 30 seconds before the page cannot be
loaded when
>>>>>>>>>> trying
>>>>>>>>>> anything proxied to Tomcat, but also accessed via
the F5 load
>>>>>>>>>> balancer.
>>>>>>>>>> Not sure where the 30 seconds comes from; perhaps
a load balancer
>>>>>>>>>> time
>>>>>>>>>> out,
>>>>>>>>>> as I dont see a "30" in my httpd configurations or
my tomcat
>>>>>>>>>> server.xml
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> You can certainly look at the Apache httpd logs,
and the tomcat
>>>>>>>>> logs, to
>>>>>>>>> see if you get a request or not.
>>>>>>>>> In Apache httpd, you can set the loglevel individually
for
>>>>>>>>> mod_proxy (if
>>>>>>>>> you are running v 2.4), and it should show something
if it gets this
>>>>>>>>> request and forwards it to tomcat.
>>>>>>>>> In tomcat, you can either enable an access log (which
will show if
>>>>>>>>> it
>>>>>>>>> receives this request), or you could temporarily remove/rename
the
>>>>>>>>> /static
>>>>>>>>> webapp. This way, it should trigger an error "not found"
which you
>>>>>>>>> would
>>>>>>>>> also see in the error log.
>>>>>>>>>
>>>>>>>>> There should be nothing between them to hinder it.  We
have many
>>>>>>>>> load
>>>>>>>>>
>>>>>>>>>> balancers and this one specifically you dont need
to open any
>>>>>>>>>> firewall
>>>>>>>>>> requests for the specific networks the HTTP servers
are on. I did
>>>>>>>>>> have to
>>>>>>>>>> get the firewall opened up to allow me to hit
>>>>>>>>>> https://loadbalancer.domain.com because the VIP for
"
>>>>>>>>>> loadbalancer.domain.com"
>>>>>>>>>> is in the DMZ, and my Desktop & VPN networks
cannot hit it on
>>>>>>>>>> 80/443
>>>>>>>>>> without opening holes.  But beyond that, any connection
from the
>>>>>>>>>> F5 to the
>>>>>>>>>> HTTP Server should be 100% open bi-directional, since
same subnet..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> But something isn't working, otherwise you would
not be asking.
>>>>>>>>> So,
>>>>>>>>>
>>>>>>>>> a) hitting the tomcat webapps through httpd seems to
be working fine
>>>>>>>>>     (browser -> httpd:23270 -> tomcat:18080 ->
webapp or static)
>>>>>>>>>
>>>>>>>>> b) hitting a non-proxied-to-tomcat resource of httpd
seems to work
>>>>>>>>> fine
>>>>>>>>> too, even through the F5
>>>>>>>>>     (browser -> F5:443 -> httpd:23270 -> html
page)
>>>>>>>>>
>>>>>>>>> c) it is only when you do :
>>>>>>>>>     (browser -> F5:443 -> httpd:23270 -> tomcat:18080
-> webapp or
>>>>>>>>> static)
>>>>>>>>>     that you see this issue
>>>>>>>>>
>>>>>>>>> It would really help if you looked in the logs of both
httpd and
>>>>>>>>> tomcat,
>>>>>>>>> and checked for differences betweens cases a, b and c
above.
>>>>>>>>>
>>>>>>>>> I believe that the F5 message with the port 23270 is
a minor issue,
>>>>>>>>> of
>>>>>>>>> information disclosure by the F5, that it should not
disclose.
>>>>>>>>>
>>>>>>>>> But the reason why it returns this error is obviously
that in that
>>>>>>>>> case,
>>>>>>>>> it does not get a response from his request to httpd.
>>>>>>>>> The reason for this response not coming back to the F5
(in case c
>>>>>>>>> only),
>>>>>>>>> can be due to either httpd or tomcat. But F5 doesn't
know about
>>>>>>>>> tomcat. So
>>>>>>>>> for the F5, it is httpd which is not responding. Thus,
>>>>>>>>> - either httpd is never getting the request from the
F5 (unlikely,
>>>>>>>>> because
>>>>>>>>> in b above it gets it and responds)
>>>>>>>>> - or httpd is getting the request from the F5, but not
forwarding
>>>>>>>>> it to
>>>>>>>>> tomcat, but also not returning an immediate error response
to F5
>>>>>>>>> (which
>>>>>>>>> seems also unlikely, because of a and b)
>>>>>>>>> - or httpd is getting the request, forwarding it to tomcat,
but not
>>>>>>>>> getting a response from tomcat. So
>>>>>>>>>     - either tomcat is never getting the request from
httpd (but in
>>>>>>>>> a, it
>>>>>>>>> gets it)
>>>>>>>>>     - or tomcat is getting the request from httpd, but
not
>>>>>>>>> responding (but
>>>>>>>>> in a, it does)
>>>>>>>>>     - or tomcat is getting the request and responding,
but the
>>>>>>>>> response
>>>>>>>>> never gets back to httpd (but in a, it does)
>>>>>>>>>
>>>>>>>>> So if a and b and c are all accurate, there is something
apparently
>>>>>>>>> illogical happening.
>>>>>>>>> This would lead to the conclusion that a and b and c
cannot all be
>>>>>>>>> accurate.
>>>>>>>>>
>>>>>>>>> The logs.. ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 21, 2017 at 2:05 PM, Antonio S. Cofino <
>>>>>>>>> cofinoa@gmail.com>
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Aaron, on tomcat instances change the redirectPort
attributte on
>>>>>>>>>> the http
>>>>>>>>>>
>>>>>>>>>>> conectó to the loabbalancer's port 443
>>>>>>>>>>>
>>>>>>>>>>> My guess is that your webapp has restriction
rule requesting SSL
>>>>>>>>>>> con
>>>>>>>>>>> fidntial channel. Therefore the non-confidential
to the 18080
>>>>>>>>>>> port from
>>>>>>>>>>> the
>>>>>>>>>>> balancer are redirected to the 23270 port, but
it should be 443.
>>>>>>>>>>>
>>>>>>>>>>> Antonio
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> El 21/2/2017 19:46, "Aaron Gray" <aaronmgray@gmail.com>
escribió:
>>>>>>>>>>>
>>>>>>>>>>> I have an application server from a vendor that
comes bundled
>>>>>>>>>>> with an
>>>>>>>>>>> additional Apache Tomcat server.  The webapp
SelfService.war is
>>>>>>>>>>> vendor
>>>>>>>>>>> supplied too.
>>>>>>>>>>>
>>>>>>>>>>> Here's my problem (IP's replaced to protect the
innocent):
>>>>>>>>>>>
>>>>>>>>>>> networks:
>>>>>>>>>>> DMZ=172.x.x.x
>>>>>>>>>>> INTERNAL=10.x.x.x
>>>>>>>>>>>
>>>>>>>>>>> server1 https listen = 172.1.1.1:23270
>>>>>>>>>>> server2 https listen = 172.1.1.2:23270
>>>>>>>>>>> F5 load balancer hostname = loadbalancer.domain.com:443
>>>>>>>>>>> backend tomcat server = 10.1.1.1:18080
>>>>>>>>>>>
>>>>>>>>>>> mod_proxy configuration:
>>>>>>>>>>> ProxyPass /SelfService http://10.1.1.1:18080/SelfService
>>>>>>>>>>> ProxyPassReverse /SelfService http://10.1.1.1:18080/SelfService
>>>>>>>>>>>
>>>>>>>>>>> When I access these DMZ webservers which mod_proxy
back to Apache
>>>>>>>>>>> Tomcat
>>>>>>>>>>> as:
>>>>>>>>>>> https://172.1.1.1:23270/SelfService
>>>>>>>>>>> and
>>>>>>>>>>> https://172.1.1.2:23270/SelfService <
>>>>>>>>>>> https://172.1.1.1:23270/SelfService
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> They load properly. Perfectly, every time!
>>>>>>>>>>>
>>>>>>>>>>> When I access these DMZ webservers via the F5
load balancer (to
>>>>>>>>>>> which I
>>>>>>>>>>> dont have access to, but the network folks configure
for me), it
>>>>>>>>>>> hangs.
>>>>>>>>>>> Eventually returns:
>>>>>>>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>>>>>>>> cant load.
>>>>>>>>>>>
>>>>>>>>>>> No idea why the URL is being re-written with
the ":23270".
>>>>>>>>>>> I added static content to the server.xml on 10.1.1.1
(Tomcat) to
>>>>>>>>>>> test:
>>>>>>>>>>> <Context docBase="/path/to/tomcat/static"
path="/static" />
>>>>>>>>>>> Then put a simple index.html in there.  Accessing
via the Apache
>>>>>>>>>>> Web
>>>>>>>>>>> Servers works fine, but if you hit it with the
Load Balancer it
>>>>>>>>>>> once
>>>>>>>>>>> again
>>>>>>>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>>>>>>>
>>>>>>>>>>> Do you have any thoughts?  Thanks so much, I
have been working
>>>>>>>>>>> with this
>>>>>>>>>>> for weeks now with no success
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> ---------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message