tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Gray <aaronmg...@gmail.com>
Subject Re: Strange URL rewrite when reverse proxy with Apache HTTP Server
Date Thu, 23 Feb 2017 17:12:38 GMT
SSLProxyEngine On
Was already turned on this entire time inside the ssl.conf (I include it)
VirtualHost section.

I am debating turning on HTTPS in Tomcat on the backend 10.x.x.x app
server, and then HTTPS the whole way through and see if that makes any
difference.  I may need to request a new firewall to be opened, which is
not able to open until 3/2.  Gonna see.


On Thu, Feb 23, 2017 at 8:22 AM, Antonio S. Cofiño <cofinoa@gmail.com>
wrote:

>
> On 23/02/17 12:43, André Warnier (tomcat) wrote:
>
>> On 22.02.2017 19:22, Aaron Gray wrote:
>>
>>> So this is interesting.
>>>
>>> So from HTTP server #1 (172.1.1.1 example) I hit:
>>> https://172.1.1.1:23270/static and I see this in the HTTP log:
>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200 32
>>> I see this in the Tomcat log:
>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 200 32
>>> ((( Yes I know they look identical, but the logformat is the same, so it
>>> comes out looking the same, but they are  two diff log files totally )))
>>>
>>> I then hit the https://loadbalancer.domain.com/static from my Win10
>>> laptop
>>>   (10.1.1.2 example)
>>> and I see this in the HTTP server #1 log, but NOTHING in the tomcat
>>> access
>>> log.
>>> 10.1.1.2 - - [22/Feb/2017:10:20:02 -0800] "-" 408 -
>>>
>>> So 408.  Timeout.  Hmm... Why?  We KNOW that it can connect from http ->
>>> tomcat:18080 perfectly.   So is it a timeout BACK to the F5? That i dont
>>> know yet.
>>>
>>
>> It looks as if, in that case, httpd is trying to connect to tomcat, but
>> - either it /is/ connecting, and sending the request, but tomcat never
>> answers
>> - or it is connecting to something else, which isn't tomcat
>> (but which accepts the request and doesn't answer either)
>>
>> I am not *sure*, but I believe, that if httpd was trying to connect to
>> something that isn't tomcat, and is not listening on port 18080, it would
>> get a different error, and a different logfile message.  So it would seem
>> that at least the TCP connection works.
>>
>
>
> Is the F5 loadbalancer configured to make reverse proxy to a SSL http?
> The directive SSLProxyEngine onthe F5 loadbalencer has to be enabled.
> http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxyengine
>
> If the loadbalancer it's delegating the SSL work to the server1
> server1 https listen = 172.1.1.1:23270
>
> then the F5 could be connecting as http regular TCP connection to a SSL
> TCP one and therefor the SSL handshake it's not happening.
>
> Antonio
>
>
>
>
>
>
>> Now there is still a puzzle there : when you connect as
>> browser --> httpd --> tomcat
>> then everything works : httpd connects to tomcat, sends a request, gets a
>> response, and returns it to the browser.
>>
>> But when you do the same as :
>> browser --> F5 --> httpd --> tomcat
>> it "does not work", and the problem seems to be at httpd --> tomat
>> Why would it make a difference in httpd --> tomcat, if it is either the
>> browser or the F5 which sends the request to httpd ?
>> the httpd --> tomcat connection should be the same, no matter what.
>>
>> Hmm, for now as puzzled as you are.
>> Maybe we have a quantum phenomenon here, where the mere fact of you
>> looking at the logfile, changes what happened beforehand. Just kidding.
>>
>> Informational note : the Apache access log line, is written when Apache
>> *has finished* processing the request (including the possible round-trip
>> through tomcat) and has sent the response to the client. (That's because,
>> for example, it can also write the time spent, and the size of the
>> response; so it has to know those before it can write the log line).
>> I presume that it is the same for tomcat.
>> Just mentioning this, so that you would maybe have a second look at the
>> tomcat access log, after a few minutes, to make sure that there really is
>> no corresponding line there.
>> It could be that tomcat actually /is/ receiving the request, and
>> processing it, but that it takes a long time, and that httpd tires of
>> waiting and already times out in the meantime. In such a case, you would
>> probably see an error some time later in the tomcat error log, when it is
>> trying to eventually return the response, and finds a closed connection
>> instead.
>> Now again, why it would do that in one case and not the other, is still a
>> mystery.
>>
>> Another note : in the "timeout" log line above, httpd is not showing the
>> request URL which triggered this timeout.
>> a) it may be showing some additional information in the Apache error log
>> b) these exists an httpd add-on module (mod_log_forensic ?) which could
>> provide more details in the log, about what happens precisely
>>
>>
>>
>>> On Tue, Feb 21, 2017 at 3:39 PM, André Warnier (tomcat) <aw@ice-sa.com>
>>> wrote:
>>>
>>> On 21.02.2017 23:28, Aaron Gray wrote:
>>>>
>>>> Antonio:  The Tomcat server has no knowledge of the F5, or that it is
>>>>> being
>>>>> fronted by an Apache HTTP Server.  I do SSL termination in Apache HTTP
>>>>> Server, and clear-text from HTTP to Tomcat.
>>>>> My redirect port for the normal HTTP listen in Tomcat is commented out.
>>>>>       <Connector port="18080" protocol="HTTP/1.1"
>>>>>                  connectionTimeout="20000" />
>>>>>       <!-- A "Connector" using the shared thread pool-->
>>>>>       <!--
>>>>>       <Connector executor="tomcatThreadPool"
>>>>>                  port="8080" protocol="HTTP/1.1"
>>>>>                  connectionTimeout="20000"
>>>>>                  redirectPort="8443" />
>>>>>       -->
>>>>>
>>>>> Andre:
>>>>> The URL I am using is https://loadbalancer.domain.com
>>>>> It is listening on port 80 and 443, if you hit 80, internally it
>>>>> redirects
>>>>> you to 443.  No SSL cert on the F5 load balancer.  It simply sends the
>>>>> traffic to one of the two HTTP servers (round-robin, also tried
>>>>> persistence, no difference).  The HTTP server is listening only for
>>>>> HTTPS
>>>>> on 23270/tcp.
>>>>> When I access these DMZ webservers via the F5 load balancer (to which
I
>>>>> dont have access to, but the network folks configure for me), it hangs.
>>>>> Eventually returns: server1 https listen = 172.1.1.1:23270
>>>>>
>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>> cant load.
>>>>>
>>>>> No idea why the URL is being re-written with the ":23270".
>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat) to test:
>>>>> <Context docBase="/path/to/tomcat/static" path="/static" />
>>>>> Then put a simple index.html in there.  Accessing via the Apache Web
>>>>> Servers works fine, but if you hit it with the Load Balancer it once
>>>>> again
>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>
>>>>> Hitting https://loadbalancer.domain.com
>>>>> I see my "Hello world!" which is all that is in index.html. This is the
>>>>> DocumentRoot of HTTP, and *not* proxied over at this time.
>>>>>
>>>>>
>>>> So in this case, there is no delay, and you get the Apache httpd-hosted
>>>> "index.html" containing "Hello World. Right ?
>>>>
>>>>    Only
>>>>
>>>> /SelfService and /static are proxied
>>>>> /static just being my test of static content, but still served up by
>>>>> Tomcat..
>>>>>
>>>>> It's exactly 30 seconds before the page cannot be loaded when trying
>>>>> anything proxied to Tomcat, but also accessed via the F5 load balancer.
>>>>> Not sure where the 30 seconds comes from; perhaps a load balancer time
>>>>> out,
>>>>> as I dont see a "30" in my httpd configurations or my tomcat server.xml
>>>>>
>>>>>
>>>>> You can certainly look at the Apache httpd logs, and the tomcat logs,
>>>> to
>>>> see if you get a request or not.
>>>> In Apache httpd, you can set the loglevel individually for mod_proxy (if
>>>> you are running v 2.4), and it should show something if it gets this
>>>> request and forwards it to tomcat.
>>>> In tomcat, you can either enable an access log (which will show if it
>>>> receives this request), or you could temporarily remove/rename the
>>>> /static
>>>> webapp. This way, it should trigger an error "not found" which you would
>>>> also see in the error log.
>>>>
>>>> There should be nothing between them to hinder it.  We have many load
>>>>
>>>>> balancers and this one specifically you dont need to open any firewall
>>>>> requests for the specific networks the HTTP servers are on. I did have
>>>>> to
>>>>> get the firewall opened up to allow me to hit
>>>>> https://loadbalancer.domain.com because the VIP for "
>>>>> loadbalancer.domain.com"
>>>>> is in the DMZ, and my Desktop & VPN networks cannot hit it on 80/443
>>>>> without opening holes.  But beyond that, any connection from the F5 to
>>>>> the
>>>>> HTTP Server should be 100% open bi-directional, since same subnet.
>>>>>
>>>>>
>>>>> But something isn't working, otherwise you would not be asking.
>>>> So,
>>>>
>>>> a) hitting the tomcat webapps through httpd seems to be working fine
>>>>    (browser -> httpd:23270 -> tomcat:18080 -> webapp or static)
>>>>
>>>> b) hitting a non-proxied-to-tomcat resource of httpd seems to work fine
>>>> too, even through the F5
>>>>    (browser -> F5:443 -> httpd:23270 -> html page)
>>>>
>>>> c) it is only when you do :
>>>>    (browser -> F5:443 -> httpd:23270 -> tomcat:18080 -> webapp
or
>>>> static)
>>>>    that you see this issue
>>>>
>>>> It would really help if you looked in the logs of both httpd and tomcat,
>>>> and checked for differences betweens cases a, b and c above.
>>>>
>>>> I believe that the F5 message with the port 23270 is a minor issue, of
>>>> information disclosure by the F5, that it should not disclose.
>>>>
>>>> But the reason why it returns this error is obviously that in that case,
>>>> it does not get a response from his request to httpd.
>>>> The reason for this response not coming back to the F5 (in case c only),
>>>> can be due to either httpd or tomcat. But F5 doesn't know about tomcat.
>>>> So
>>>> for the F5, it is httpd which is not responding. Thus,
>>>> - either httpd is never getting the request from the F5 (unlikely,
>>>> because
>>>> in b above it gets it and responds)
>>>> - or httpd is getting the request from the F5, but not forwarding it to
>>>> tomcat, but also not returning an immediate error response to F5 (which
>>>> seems also unlikely, because of a and b)
>>>> - or httpd is getting the request, forwarding it to tomcat, but not
>>>> getting a response from tomcat. So
>>>>    - either tomcat is never getting the request from httpd (but in a, it
>>>> gets it)
>>>>    - or tomcat is getting the request from httpd, but not responding
>>>> (but
>>>> in a, it does)
>>>>    - or tomcat is getting the request and responding, but the response
>>>> never gets back to httpd (but in a, it does)
>>>>
>>>> So if a and b and c are all accurate, there is something apparently
>>>> illogical happening.
>>>> This would lead to the conclusion that a and b and c cannot all be
>>>> accurate.
>>>>
>>>> The logs.. ?
>>>>
>>>>
>>>>
>>>> On Tue, Feb 21, 2017 at 2:05 PM, Antonio S. Cofino <cofinoa@gmail.com>
>>>>
>>>>> wrote:
>>>>>
>>>>> Aaron, on tomcat instances change the redirectPort attributte on the
>>>>> http
>>>>>
>>>>>> conectó to the loabbalancer's port 443
>>>>>>
>>>>>> My guess is that your webapp has restriction rule requesting SSL
con
>>>>>> fidntial channel. Therefore the non-confidential to the 18080 port
>>>>>> from
>>>>>> the
>>>>>> balancer are redirected to the 23270 port, but it should be 443.
>>>>>>
>>>>>> Antonio
>>>>>>
>>>>>>
>>>>>>
>>>>>> El 21/2/2017 19:46, "Aaron Gray" <aaronmgray@gmail.com> escribió:
>>>>>>
>>>>>> I have an application server from a vendor that comes bundled with
an
>>>>>> additional Apache Tomcat server.  The webapp SelfService.war is vendor
>>>>>> supplied too.
>>>>>>
>>>>>> Here's my problem (IP's replaced to protect the innocent):
>>>>>>
>>>>>> networks:
>>>>>> DMZ=172.x.x.x
>>>>>> INTERNAL=10.x.x.x
>>>>>>
>>>>>> server1 https listen = 172.1.1.1:23270
>>>>>> server2 https listen = 172.1.1.2:23270
>>>>>> F5 load balancer hostname = loadbalancer.domain.com:443
>>>>>> backend tomcat server = 10.1.1.1:18080
>>>>>>
>>>>>> mod_proxy configuration:
>>>>>> ProxyPass /SelfService http://10.1.1.1:18080/SelfService
>>>>>> ProxyPassReverse /SelfService http://10.1.1.1:18080/SelfService
>>>>>>
>>>>>> When I access these DMZ webservers which mod_proxy back to Apache
>>>>>> Tomcat
>>>>>> as:
>>>>>> https://172.1.1.1:23270/SelfService
>>>>>> and
>>>>>> https://172.1.1.2:23270/SelfService <https://172.1.1.1:23270/SelfS
>>>>>> ervice
>>>>>>
>>>>>>>
>>>>>>> They load properly. Perfectly, every time!
>>>>>>
>>>>>> When I access these DMZ webservers via the F5 load balancer (to which
>>>>>> I
>>>>>> dont have access to, but the network folks configure for me), it
>>>>>> hangs.
>>>>>> Eventually returns:
>>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>>> cant load.
>>>>>>
>>>>>> No idea why the URL is being re-written with the ":23270".
>>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat) to
test:
>>>>>> <Context docBase="/path/to/tomcat/static" path="/static" />
>>>>>> Then put a simple index.html in there.  Accessing via the Apache
Web
>>>>>> Servers works fine, but if you hit it with the Load Balancer it once
>>>>>> again
>>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>>
>>>>>> Do you have any thoughts?  Thanks so much, I have been working with
>>>>>> this
>>>>>> for weeks now with no success
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message