tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antonio S. Cofiño <cofi...@gmail.com>
Subject Re: Strange URL rewrite when reverse proxy with Apache HTTP Server
Date Thu, 23 Feb 2017 16:22:47 GMT

On 23/02/17 12:43, André Warnier (tomcat) wrote:
> On 22.02.2017 19:22, Aaron Gray wrote:
>> So this is interesting.
>>
>> So from HTTP server #1 (172.1.1.1 example) I hit:
>> https://172.1.1.1:23270/static and I see this in the HTTP log:
>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 
>> 200 32
>> I see this in the Tomcat log:
>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1" 
>> 200 32
>> ((( Yes I know they look identical, but the logformat is the same, so it
>> comes out looking the same, but they are  two diff log files totally )))
>>
>> I then hit the https://loadbalancer.domain.com/static from my Win10 
>> laptop
>>   (10.1.1.2 example)
>> and I see this in the HTTP server #1 log, but NOTHING in the tomcat 
>> access
>> log.
>> 10.1.1.2 - - [22/Feb/2017:10:20:02 -0800] "-" 408 -
>>
>> So 408.  Timeout.  Hmm... Why?  We KNOW that it can connect from http ->
>> tomcat:18080 perfectly.   So is it a timeout BACK to the F5? That i dont
>> know yet.
>
> It looks as if, in that case, httpd is trying to connect to tomcat, but
> - either it /is/ connecting, and sending the request, but tomcat never 
> answers
> - or it is connecting to something else, which isn't tomcat
> (but which accepts the request and doesn't answer either)
>
> I am not *sure*, but I believe, that if httpd was trying to connect to 
> something that isn't tomcat, and is not listening on port 18080, it 
> would get a different error, and a different logfile message.  So it 
> would seem that at least the TCP connection works.


Is the F5 loadbalancer configured to make reverse proxy to a SSL http?
The directive SSLProxyEngine onthe F5 loadbalencer has to be enabled.
http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxyengine

If the loadbalancer it's delegating the SSL work to the server1
server1 https listen = 172.1.1.1:23270

then the F5 could be connecting as http regular TCP connection to a SSL 
TCP one and therefor the SSL handshake it's not happening.

Antonio





>
> Now there is still a puzzle there : when you connect as
> browser --> httpd --> tomcat
> then everything works : httpd connects to tomcat, sends a request, 
> gets a response, and returns it to the browser.
>
> But when you do the same as :
> browser --> F5 --> httpd --> tomcat
> it "does not work", and the problem seems to be at httpd --> tomat
> Why would it make a difference in httpd --> tomcat, if it is either 
> the browser or the F5 which sends the request to httpd ?
> the httpd --> tomcat connection should be the same, no matter what.
>
> Hmm, for now as puzzled as you are.
> Maybe we have a quantum phenomenon here, where the mere fact of you 
> looking at the logfile, changes what happened beforehand. Just kidding.
>
> Informational note : the Apache access log line, is written when 
> Apache *has finished* processing the request (including the possible 
> round-trip through tomcat) and has sent the response to the client. 
> (That's because, for example, it can also write the time spent, and 
> the size of the response; so it has to know those before it can write 
> the log line).
> I presume that it is the same for tomcat.
> Just mentioning this, so that you would maybe have a second look at 
> the tomcat access log, after a few minutes, to make sure that there 
> really is no corresponding line there.
> It could be that tomcat actually /is/ receiving the request, and 
> processing it, but that it takes a long time, and that httpd tires of 
> waiting and already times out in the meantime. In such a case, you 
> would probably see an error some time later in the tomcat error log, 
> when it is trying to eventually return the response, and finds a 
> closed connection instead.
> Now again, why it would do that in one case and not the other, is 
> still a mystery.
>
> Another note : in the "timeout" log line above, httpd is not showing 
> the request URL which triggered this timeout.
> a) it may be showing some additional information in the Apache error log
> b) these exists an httpd add-on module (mod_log_forensic ?) which 
> could provide more details in the log, about what happens precisely
>
>
>>
>> On Tue, Feb 21, 2017 at 3:39 PM, André Warnier (tomcat) <aw@ice-sa.com>
>> wrote:
>>
>>> On 21.02.2017 23:28, Aaron Gray wrote:
>>>
>>>> Antonio:  The Tomcat server has no knowledge of the F5, or that it is
>>>> being
>>>> fronted by an Apache HTTP Server.  I do SSL termination in Apache HTTP
>>>> Server, and clear-text from HTTP to Tomcat.
>>>> My redirect port for the normal HTTP listen in Tomcat is commented 
>>>> out.
>>>>       <Connector port="18080" protocol="HTTP/1.1"
>>>>                  connectionTimeout="20000" />
>>>>       <!-- A "Connector" using the shared thread pool-->
>>>>       <!--
>>>>       <Connector executor="tomcatThreadPool"
>>>>                  port="8080" protocol="HTTP/1.1"
>>>>                  connectionTimeout="20000"
>>>>                  redirectPort="8443" />
>>>>       -->
>>>>
>>>> Andre:
>>>> The URL I am using is https://loadbalancer.domain.com
>>>> It is listening on port 80 and 443, if you hit 80, internally it 
>>>> redirects
>>>> you to 443.  No SSL cert on the F5 load balancer.  It simply sends the
>>>> traffic to one of the two HTTP servers (round-robin, also tried
>>>> persistence, no difference).  The HTTP server is listening only for 
>>>> HTTPS
>>>> on 23270/tcp.
>>>> When I access these DMZ webservers via the F5 load balancer (to 
>>>> which I
>>>> dont have access to, but the network folks configure for me), it 
>>>> hangs.
>>>> Eventually returns: server1 https listen = 172.1.1.1:23270
>>>> https://loadbalancer.domain.com:23270/SelfService
>>>> cant load.
>>>>
>>>> No idea why the URL is being re-written with the ":23270".
>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat) to test:
>>>> <Context docBase="/path/to/tomcat/static" path="/static" />
>>>> Then put a simple index.html in there.  Accessing via the Apache Web
>>>> Servers works fine, but if you hit it with the Load Balancer it once
>>>> again
>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>
>>>> Hitting https://loadbalancer.domain.com
>>>> I see my "Hello world!" which is all that is in index.html. This is 
>>>> the
>>>> DocumentRoot of HTTP, and *not* proxied over at this time.
>>>>
>>>
>>> So in this case, there is no delay, and you get the Apache httpd-hosted
>>> "index.html" containing "Hello World. Right ?
>>>
>>>    Only
>>>
>>>> /SelfService and /static are proxied
>>>> /static just being my test of static content, but still served up by
>>>> Tomcat..
>>>>
>>>> It's exactly 30 seconds before the page cannot be loaded when trying
>>>> anything proxied to Tomcat, but also accessed via the F5 load 
>>>> balancer.
>>>> Not sure where the 30 seconds comes from; perhaps a load balancer time
>>>> out,
>>>> as I dont see a "30" in my httpd configurations or my tomcat 
>>>> server.xml
>>>>
>>>>
>>> You can certainly look at the Apache httpd logs, and the tomcat 
>>> logs, to
>>> see if you get a request or not.
>>> In Apache httpd, you can set the loglevel individually for mod_proxy 
>>> (if
>>> you are running v 2.4), and it should show something if it gets this
>>> request and forwards it to tomcat.
>>> In tomcat, you can either enable an access log (which will show if it
>>> receives this request), or you could temporarily remove/rename the 
>>> /static
>>> webapp. This way, it should trigger an error "not found" which you 
>>> would
>>> also see in the error log.
>>>
>>> There should be nothing between them to hinder it.  We have many load
>>>> balancers and this one specifically you dont need to open any firewall
>>>> requests for the specific networks the HTTP servers are on. I did 
>>>> have to
>>>> get the firewall opened up to allow me to hit
>>>> https://loadbalancer.domain.com because the VIP for "
>>>> loadbalancer.domain.com"
>>>> is in the DMZ, and my Desktop & VPN networks cannot hit it on 80/443
>>>> without opening holes.  But beyond that, any connection from the F5 
>>>> to the
>>>> HTTP Server should be 100% open bi-directional, since same subnet.
>>>>
>>>>
>>> But something isn't working, otherwise you would not be asking.
>>> So,
>>>
>>> a) hitting the tomcat webapps through httpd seems to be working fine
>>>    (browser -> httpd:23270 -> tomcat:18080 -> webapp or static)
>>>
>>> b) hitting a non-proxied-to-tomcat resource of httpd seems to work fine
>>> too, even through the F5
>>>    (browser -> F5:443 -> httpd:23270 -> html page)
>>>
>>> c) it is only when you do :
>>>    (browser -> F5:443 -> httpd:23270 -> tomcat:18080 -> webapp or

>>> static)
>>>    that you see this issue
>>>
>>> It would really help if you looked in the logs of both httpd and 
>>> tomcat,
>>> and checked for differences betweens cases a, b and c above.
>>>
>>> I believe that the F5 message with the port 23270 is a minor issue, of
>>> information disclosure by the F5, that it should not disclose.
>>>
>>> But the reason why it returns this error is obviously that in that 
>>> case,
>>> it does not get a response from his request to httpd.
>>> The reason for this response not coming back to the F5 (in case c 
>>> only),
>>> can be due to either httpd or tomcat. But F5 doesn't know about 
>>> tomcat. So
>>> for the F5, it is httpd which is not responding. Thus,
>>> - either httpd is never getting the request from the F5 (unlikely, 
>>> because
>>> in b above it gets it and responds)
>>> - or httpd is getting the request from the F5, but not forwarding it to
>>> tomcat, but also not returning an immediate error response to F5 (which
>>> seems also unlikely, because of a and b)
>>> - or httpd is getting the request, forwarding it to tomcat, but not
>>> getting a response from tomcat. So
>>>    - either tomcat is never getting the request from httpd (but in 
>>> a, it
>>> gets it)
>>>    - or tomcat is getting the request from httpd, but not responding 
>>> (but
>>> in a, it does)
>>>    - or tomcat is getting the request and responding, but the response
>>> never gets back to httpd (but in a, it does)
>>>
>>> So if a and b and c are all accurate, there is something apparently
>>> illogical happening.
>>> This would lead to the conclusion that a and b and c cannot all be
>>> accurate.
>>>
>>> The logs.. ?
>>>
>>>
>>>
>>> On Tue, Feb 21, 2017 at 2:05 PM, Antonio S. Cofino <cofinoa@gmail.com>
>>>> wrote:
>>>>
>>>> Aaron, on tomcat instances change the redirectPort attributte on 
>>>> the http
>>>>> conectó to the loabbalancer's port 443
>>>>>
>>>>> My guess is that your webapp has restriction rule requesting SSL con
>>>>> fidntial channel. Therefore the non-confidential to the 18080 port 
>>>>> from
>>>>> the
>>>>> balancer are redirected to the 23270 port, but it should be 443.
>>>>>
>>>>> Antonio
>>>>>
>>>>>
>>>>>
>>>>> El 21/2/2017 19:46, "Aaron Gray" <aaronmgray@gmail.com> escribió:
>>>>>
>>>>> I have an application server from a vendor that comes bundled with an
>>>>> additional Apache Tomcat server.  The webapp SelfService.war is 
>>>>> vendor
>>>>> supplied too.
>>>>>
>>>>> Here's my problem (IP's replaced to protect the innocent):
>>>>>
>>>>> networks:
>>>>> DMZ=172.x.x.x
>>>>> INTERNAL=10.x.x.x
>>>>>
>>>>> server1 https listen = 172.1.1.1:23270
>>>>> server2 https listen = 172.1.1.2:23270
>>>>> F5 load balancer hostname = loadbalancer.domain.com:443
>>>>> backend tomcat server = 10.1.1.1:18080
>>>>>
>>>>> mod_proxy configuration:
>>>>> ProxyPass /SelfService http://10.1.1.1:18080/SelfService
>>>>> ProxyPassReverse /SelfService http://10.1.1.1:18080/SelfService
>>>>>
>>>>> When I access these DMZ webservers which mod_proxy back to Apache 
>>>>> Tomcat
>>>>> as:
>>>>> https://172.1.1.1:23270/SelfService
>>>>> and
>>>>> https://172.1.1.2:23270/SelfService 
>>>>> <https://172.1.1.1:23270/SelfService
>>>>>>
>>>>> They load properly. Perfectly, every time!
>>>>>
>>>>> When I access these DMZ webservers via the F5 load balancer (to 
>>>>> which I
>>>>> dont have access to, but the network folks configure for me), it 
>>>>> hangs.
>>>>> Eventually returns:
>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>> cant load.
>>>>>
>>>>> No idea why the URL is being re-written with the ":23270".
>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat) to 
>>>>> test:
>>>>> <Context docBase="/path/to/tomcat/static" path="/static" />
>>>>> Then put a simple index.html in there.  Accessing via the Apache Web
>>>>> Servers works fine, but if you hit it with the Load Balancer it once
>>>>> again
>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>
>>>>> Do you have any thoughts?  Thanks so much, I have been working 
>>>>> with this
>>>>> for weeks now with no success
>>>>>
>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message