tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Gray <aaronmg...@gmail.com>
Subject Re: Strange URL rewrite when reverse proxy with Apache HTTP Server
Date Thu, 23 Feb 2017 17:22:17 GMT
Working with my F5 guy, we had an idea, since 80/tcp and 443/tcp were
already open to the VIP on the F5, we simply turned of 80 -> 443 redirect
on the F5, and then configured the F5 to use the non-HTTP port in Apache
HTTP Server (the two backend servers).  So its HTTP the entire way through,
and the same issue persisted.

browser -> F5(http) -> httpd(http port) -> tomcat(http port, nonssl)

So i think i can rule out any possibility that the fact I was terminating
SSL in Apache HTTP Server then proxying clear-text back as a reason.  It
worked fine like I said if you just go direct to the HTTP Server and slip
the F5.   And F5 works landing the /healthcheck.html (HTTP KeepAlive in the
root, which isnt proxied), so all good there.  Just cant /SelfService
(configured w/ Proxy)

thanks for your support here though.

On Thu, Feb 23, 2017 at 9:12 AM, Aaron Gray <aaronmgray@gmail.com> wrote:

> SSLProxyEngine On
> Was already turned on this entire time inside the ssl.conf (I include it)
> VirtualHost section.
>
> I am debating turning on HTTPS in Tomcat on the backend 10.x.x.x app
> server, and then HTTPS the whole way through and see if that makes any
> difference.  I may need to request a new firewall to be opened, which is
> not able to open until 3/2.  Gonna see.
>
>
> On Thu, Feb 23, 2017 at 8:22 AM, Antonio S. Cofiño <cofinoa@gmail.com>
> wrote:
>
>>
>> On 23/02/17 12:43, André Warnier (tomcat) wrote:
>>
>>> On 22.02.2017 19:22, Aaron Gray wrote:
>>>
>>>> So this is interesting.
>>>>
>>>> So from HTTP server #1 (172.1.1.1 example) I hit:
>>>> https://172.1.1.1:23270/static and I see this in the HTTP log:
>>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200
>>>> 32
>>>> I see this in the Tomcat log:
>>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200
>>>> 32
>>>> ((( Yes I know they look identical, but the logformat is the same, so it
>>>> comes out looking the same, but they are  two diff log files totally )))
>>>>
>>>> I then hit the https://loadbalancer.domain.com/static from my Win10
>>>> laptop
>>>>   (10.1.1.2 example)
>>>> and I see this in the HTTP server #1 log, but NOTHING in the tomcat
>>>> access
>>>> log.
>>>> 10.1.1.2 - - [22/Feb/2017:10:20:02 -0800] "-" 408 -
>>>>
>>>> So 408.  Timeout.  Hmm... Why?  We KNOW that it can connect from http ->
>>>> tomcat:18080 perfectly.   So is it a timeout BACK to the F5? That i dont
>>>> know yet.
>>>>
>>>
>>> It looks as if, in that case, httpd is trying to connect to tomcat, but
>>> - either it /is/ connecting, and sending the request, but tomcat never
>>> answers
>>> - or it is connecting to something else, which isn't tomcat
>>> (but which accepts the request and doesn't answer either)
>>>
>>> I am not *sure*, but I believe, that if httpd was trying to connect to
>>> something that isn't tomcat, and is not listening on port 18080, it would
>>> get a different error, and a different logfile message.  So it would seem
>>> that at least the TCP connection works.
>>>
>>
>>
>> Is the F5 loadbalancer configured to make reverse proxy to a SSL http?
>> The directive SSLProxyEngine onthe F5 loadbalencer has to be enabled.
>> http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxyengine
>>
>> If the loadbalancer it's delegating the SSL work to the server1
>> server1 https listen = 172.1.1.1:23270
>>
>> then the F5 could be connecting as http regular TCP connection to a SSL
>> TCP one and therefor the SSL handshake it's not happening.
>>
>> Antonio
>>
>>
>>
>>
>>
>>
>>> Now there is still a puzzle there : when you connect as
>>> browser --> httpd --> tomcat
>>> then everything works : httpd connects to tomcat, sends a request, gets
>>> a response, and returns it to the browser.
>>>
>>> But when you do the same as :
>>> browser --> F5 --> httpd --> tomcat
>>> it "does not work", and the problem seems to be at httpd --> tomat
>>> Why would it make a difference in httpd --> tomcat, if it is either the
>>> browser or the F5 which sends the request to httpd ?
>>> the httpd --> tomcat connection should be the same, no matter what.
>>>
>>> Hmm, for now as puzzled as you are.
>>> Maybe we have a quantum phenomenon here, where the mere fact of you
>>> looking at the logfile, changes what happened beforehand. Just kidding.
>>>
>>> Informational note : the Apache access log line, is written when Apache
>>> *has finished* processing the request (including the possible round-trip
>>> through tomcat) and has sent the response to the client. (That's because,
>>> for example, it can also write the time spent, and the size of the
>>> response; so it has to know those before it can write the log line).
>>> I presume that it is the same for tomcat.
>>> Just mentioning this, so that you would maybe have a second look at the
>>> tomcat access log, after a few minutes, to make sure that there really is
>>> no corresponding line there.
>>> It could be that tomcat actually /is/ receiving the request, and
>>> processing it, but that it takes a long time, and that httpd tires of
>>> waiting and already times out in the meantime. In such a case, you would
>>> probably see an error some time later in the tomcat error log, when it is
>>> trying to eventually return the response, and finds a closed connection
>>> instead.
>>> Now again, why it would do that in one case and not the other, is still
>>> a mystery.
>>>
>>> Another note : in the "timeout" log line above, httpd is not showing the
>>> request URL which triggered this timeout.
>>> a) it may be showing some additional information in the Apache error log
>>> b) these exists an httpd add-on module (mod_log_forensic ?) which could
>>> provide more details in the log, about what happens precisely
>>>
>>>
>>>
>>>> On Tue, Feb 21, 2017 at 3:39 PM, André Warnier (tomcat) <aw@ice-sa.com>
>>>> wrote:
>>>>
>>>> On 21.02.2017 23:28, Aaron Gray wrote:
>>>>>
>>>>> Antonio:  The Tomcat server has no knowledge of the F5, or that it is
>>>>>> being
>>>>>> fronted by an Apache HTTP Server.  I do SSL termination in Apache
HTTP
>>>>>> Server, and clear-text from HTTP to Tomcat.
>>>>>> My redirect port for the normal HTTP listen in Tomcat is commented
>>>>>> out.
>>>>>>       <Connector port="18080" protocol="HTTP/1.1"
>>>>>>                  connectionTimeout="20000" />
>>>>>>       <!-- A "Connector" using the shared thread pool-->
>>>>>>       <!--
>>>>>>       <Connector executor="tomcatThreadPool"
>>>>>>                  port="8080" protocol="HTTP/1.1"
>>>>>>                  connectionTimeout="20000"
>>>>>>                  redirectPort="8443" />
>>>>>>       -->
>>>>>>
>>>>>> Andre:
>>>>>> The URL I am using is https://loadbalancer.domain.com
>>>>>> It is listening on port 80 and 443, if you hit 80, internally it
>>>>>> redirects
>>>>>> you to 443.  No SSL cert on the F5 load balancer.  It simply sends
the
>>>>>> traffic to one of the two HTTP servers (round-robin, also tried
>>>>>> persistence, no difference).  The HTTP server is listening only for
>>>>>> HTTPS
>>>>>> on 23270/tcp.
>>>>>> When I access these DMZ webservers via the F5 load balancer (to which
>>>>>> I
>>>>>> dont have access to, but the network folks configure for me), it
>>>>>> hangs.
>>>>>> Eventually returns: server1 https listen = 172.1.1.1:23270
>>>>>>
>>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>>> cant load.
>>>>>>
>>>>>> No idea why the URL is being re-written with the ":23270".
>>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat) to
test:
>>>>>> <Context docBase="/path/to/tomcat/static" path="/static" />
>>>>>> Then put a simple index.html in there.  Accessing via the Apache
Web
>>>>>> Servers works fine, but if you hit it with the Load Balancer it once
>>>>>> again
>>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>>
>>>>>> Hitting https://loadbalancer.domain.com
>>>>>> I see my "Hello world!" which is all that is in index.html. This
is
>>>>>> the
>>>>>> DocumentRoot of HTTP, and *not* proxied over at this time.
>>>>>>
>>>>>>
>>>>> So in this case, there is no delay, and you get the Apache httpd-hosted
>>>>> "index.html" containing "Hello World. Right ?
>>>>>
>>>>>    Only
>>>>>
>>>>> /SelfService and /static are proxied
>>>>>> /static just being my test of static content, but still served up
by
>>>>>> Tomcat..
>>>>>>
>>>>>> It's exactly 30 seconds before the page cannot be loaded when trying
>>>>>> anything proxied to Tomcat, but also accessed via the F5 load
>>>>>> balancer.
>>>>>> Not sure where the 30 seconds comes from; perhaps a load balancer
time
>>>>>> out,
>>>>>> as I dont see a "30" in my httpd configurations or my tomcat
>>>>>> server.xml
>>>>>>
>>>>>>
>>>>>> You can certainly look at the Apache httpd logs, and the tomcat logs,
>>>>> to
>>>>> see if you get a request or not.
>>>>> In Apache httpd, you can set the loglevel individually for mod_proxy
>>>>> (if
>>>>> you are running v 2.4), and it should show something if it gets this
>>>>> request and forwards it to tomcat.
>>>>> In tomcat, you can either enable an access log (which will show if it
>>>>> receives this request), or you could temporarily remove/rename the
>>>>> /static
>>>>> webapp. This way, it should trigger an error "not found" which you
>>>>> would
>>>>> also see in the error log.
>>>>>
>>>>> There should be nothing between them to hinder it.  We have many load
>>>>>
>>>>>> balancers and this one specifically you dont need to open any firewall
>>>>>> requests for the specific networks the HTTP servers are on. I did
>>>>>> have to
>>>>>> get the firewall opened up to allow me to hit
>>>>>> https://loadbalancer.domain.com because the VIP for "
>>>>>> loadbalancer.domain.com"
>>>>>> is in the DMZ, and my Desktop & VPN networks cannot hit it on
80/443
>>>>>> without opening holes.  But beyond that, any connection from the
F5
>>>>>> to the
>>>>>> HTTP Server should be 100% open bi-directional, since same subnet.
>>>>>>
>>>>>>
>>>>>> But something isn't working, otherwise you would not be asking.
>>>>> So,
>>>>>
>>>>> a) hitting the tomcat webapps through httpd seems to be working fine
>>>>>    (browser -> httpd:23270 -> tomcat:18080 -> webapp or static)
>>>>>
>>>>> b) hitting a non-proxied-to-tomcat resource of httpd seems to work fine
>>>>> too, even through the F5
>>>>>    (browser -> F5:443 -> httpd:23270 -> html page)
>>>>>
>>>>> c) it is only when you do :
>>>>>    (browser -> F5:443 -> httpd:23270 -> tomcat:18080 -> webapp
or
>>>>> static)
>>>>>    that you see this issue
>>>>>
>>>>> It would really help if you looked in the logs of both httpd and
>>>>> tomcat,
>>>>> and checked for differences betweens cases a, b and c above.
>>>>>
>>>>> I believe that the F5 message with the port 23270 is a minor issue, of
>>>>> information disclosure by the F5, that it should not disclose.
>>>>>
>>>>> But the reason why it returns this error is obviously that in that
>>>>> case,
>>>>> it does not get a response from his request to httpd.
>>>>> The reason for this response not coming back to the F5 (in case c
>>>>> only),
>>>>> can be due to either httpd or tomcat. But F5 doesn't know about
>>>>> tomcat. So
>>>>> for the F5, it is httpd which is not responding. Thus,
>>>>> - either httpd is never getting the request from the F5 (unlikely,
>>>>> because
>>>>> in b above it gets it and responds)
>>>>> - or httpd is getting the request from the F5, but not forwarding it
to
>>>>> tomcat, but also not returning an immediate error response to F5 (which
>>>>> seems also unlikely, because of a and b)
>>>>> - or httpd is getting the request, forwarding it to tomcat, but not
>>>>> getting a response from tomcat. So
>>>>>    - either tomcat is never getting the request from httpd (but in a,
>>>>> it
>>>>> gets it)
>>>>>    - or tomcat is getting the request from httpd, but not responding
>>>>> (but
>>>>> in a, it does)
>>>>>    - or tomcat is getting the request and responding, but the response
>>>>> never gets back to httpd (but in a, it does)
>>>>>
>>>>> So if a and b and c are all accurate, there is something apparently
>>>>> illogical happening.
>>>>> This would lead to the conclusion that a and b and c cannot all be
>>>>> accurate.
>>>>>
>>>>> The logs.. ?
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 21, 2017 at 2:05 PM, Antonio S. Cofino <cofinoa@gmail.com>
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Aaron, on tomcat instances change the redirectPort attributte on
the
>>>>>> http
>>>>>>
>>>>>>> conectó to the loabbalancer's port 443
>>>>>>>
>>>>>>> My guess is that your webapp has restriction rule requesting
SSL con
>>>>>>> fidntial channel. Therefore the non-confidential to the 18080
port
>>>>>>> from
>>>>>>> the
>>>>>>> balancer are redirected to the 23270 port, but it should be 443.
>>>>>>>
>>>>>>> Antonio
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> El 21/2/2017 19:46, "Aaron Gray" <aaronmgray@gmail.com>
escribió:
>>>>>>>
>>>>>>> I have an application server from a vendor that comes bundled
with an
>>>>>>> additional Apache Tomcat server.  The webapp SelfService.war
is
>>>>>>> vendor
>>>>>>> supplied too.
>>>>>>>
>>>>>>> Here's my problem (IP's replaced to protect the innocent):
>>>>>>>
>>>>>>> networks:
>>>>>>> DMZ=172.x.x.x
>>>>>>> INTERNAL=10.x.x.x
>>>>>>>
>>>>>>> server1 https listen = 172.1.1.1:23270
>>>>>>> server2 https listen = 172.1.1.2:23270
>>>>>>> F5 load balancer hostname = loadbalancer.domain.com:443
>>>>>>> backend tomcat server = 10.1.1.1:18080
>>>>>>>
>>>>>>> mod_proxy configuration:
>>>>>>> ProxyPass /SelfService http://10.1.1.1:18080/SelfService
>>>>>>> ProxyPassReverse /SelfService http://10.1.1.1:18080/SelfService
>>>>>>>
>>>>>>> When I access these DMZ webservers which mod_proxy back to Apache
>>>>>>> Tomcat
>>>>>>> as:
>>>>>>> https://172.1.1.1:23270/SelfService
>>>>>>> and
>>>>>>> https://172.1.1.2:23270/SelfService <https://172.1.1.1:23270/SelfS
>>>>>>> ervice
>>>>>>>
>>>>>>>>
>>>>>>>> They load properly. Perfectly, every time!
>>>>>>>
>>>>>>> When I access these DMZ webservers via the F5 load balancer (to
>>>>>>> which I
>>>>>>> dont have access to, but the network folks configure for me),
it
>>>>>>> hangs.
>>>>>>> Eventually returns:
>>>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>>>> cant load.
>>>>>>>
>>>>>>> No idea why the URL is being re-written with the ":23270".
>>>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat)
to
>>>>>>> test:
>>>>>>> <Context docBase="/path/to/tomcat/static" path="/static" />
>>>>>>> Then put a simple index.html in there.  Accessing via the Apache
Web
>>>>>>> Servers works fine, but if you hit it with the Load Balancer
it once
>>>>>>> again
>>>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>>>
>>>>>>> Do you have any thoughts?  Thanks so much, I have been working
with
>>>>>>> this
>>>>>>> for weeks now with no success
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message