tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Warnier (tomcat) ...@ice-sa.com>
Subject Re: Strange URL rewrite when reverse proxy with Apache HTTP Server
Date Thu, 23 Feb 2017 22:57:30 GMT
On 23.02.2017 19:10, Aaron Gray wrote:
> Okay, guys, please accept my most sincere apology on this thing.  I just
> figured 1 thing out.
>
> browser -> f5:443 -> httpd (23270, https) -> tomcat (http)
> using the /static works! (see below)
>
> If /static/index.html wasnt specified, then it hangs, then comes back as
> https://loadbalancer.domain.com:23270/static
> But if you go https://loadbalancer.domain.com/static/index.html it works!!!
>
> DirectoryIndex index.html
> is present.  because if you just go to https://webserver1:23270/static
> it works fine, no need for the extra /index.html
>

Something vague in my mind tells me that the problem is probably right there, in the 
Redirect response, but it's quite late here in Europe now, and I can't quite figure it out

anymore..

> not sure why that is different there.
>
> I only found this out using curl and just happened to think of it.
> So you see the difference, from the tomcat access log, in the two
> variations.
>
> 172.18.160.66 - - [23/Feb/2017:10:00:48 -0800] "GET /static HTTP/1.1" 302 -
> 172.18.160.66 - - [23/Feb/2017:10:03:04 -0800] "GET /static/index.html
> HTTP/1.1" 200 32

You must be very selectively filtering this log, because :
- the first response above is a Redirect response, normal because of the DirectoryIndex
- the second response /could/ be the follow-up request that the browser makes after this 
Redirect, /except/ that it seems to happen almost 3 minutes later, which would mean either

a /very/ slow connection, or a /very/ slow browser.

>
> Still /SelfService wont work though
> I cant think of a way to do /SelfService/something...

neither should you need to, unless this is also affected by the Redirect syndrom.

> Since it is a web app and i am no developer and its all from the vendor
> (BMC Software) and it never shows naything other than /SelfService in the
> URL.  I dont think its a frame, but once again i dont know these things.  I
> know they told me it is HTML5, as it was using Silverlight back in the
> older release.

None of this should matter, really.

What about these Apache and tomcat *error logs* ? what do they tell you, when the initial

problem occurs ?

>
>
> On Thu, Feb 23, 2017 at 9:22 AM, Aaron Gray <aaronmgray@gmail.com> wrote:
>
>> Working with my F5 guy, we had an idea, since 80/tcp and 443/tcp were
>> already open to the VIP on the F5, we simply turned of 80 -> 443 redirect
>> on the F5, and then configured the F5 to use the non-HTTP port in Apache
>> HTTP Server (the two backend servers).  So its HTTP the entire way through,
>> and the same issue persisted.
>>
>> browser -> F5(http) -> httpd(http port) -> tomcat(http port, nonssl)
>>
>> So i think i can rule out any possibility that the fact I was terminating
>> SSL in Apache HTTP Server then proxying clear-text back as a reason.  It
>> worked fine like I said if you just go direct to the HTTP Server and slip
>> the F5.   And F5 works landing the /healthcheck.html (HTTP KeepAlive in the
>> root, which isnt proxied), so all good there.  Just cant /SelfService
>> (configured w/ Proxy)
>>
>> thanks for your support here though.
>>
>> On Thu, Feb 23, 2017 at 9:12 AM, Aaron Gray <aaronmgray@gmail.com> wrote:
>>
>>> SSLProxyEngine On
>>> Was already turned on this entire time inside the ssl.conf (I include it)
>>> VirtualHost section.
>>>
>>> I am debating turning on HTTPS in Tomcat on the backend 10.x.x.x app
>>> server, and then HTTPS the whole way through and see if that makes any
>>> difference.  I may need to request a new firewall to be opened, which is
>>> not able to open until 3/2.  Gonna see.
>>>
>>>
>>> On Thu, Feb 23, 2017 at 8:22 AM, Antonio S. Cofiño <cofinoa@gmail.com>
>>> wrote:
>>>
>>>>
>>>> On 23/02/17 12:43, André Warnier (tomcat) wrote:
>>>>
>>>>> On 22.02.2017 19:22, Aaron Gray wrote:
>>>>>
>>>>>> So this is interesting.
>>>>>>
>>>>>> So from HTTP server #1 (172.1.1.1 example) I hit:
>>>>>> https://172.1.1.1:23270/static and I see this in the HTTP log:
>>>>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1"
200
>>>>>> 32
>>>>>> I see this in the Tomcat log:
>>>>>> 172.1.1.1 - - [22/Feb/2017:10:14:48 -0800] "GET /static/ HTTP/1.1"
200
>>>>>> 32
>>>>>> ((( Yes I know they look identical, but the logformat is the same,
so
>>>>>> it
>>>>>> comes out looking the same, but they are  two diff log files totally
>>>>>> )))
>>>>>>
>>>>>> I then hit the https://loadbalancer.domain.com/static from my Win10
>>>>>> laptop
>>>>>>    (10.1.1.2 example)
>>>>>> and I see this in the HTTP server #1 log, but NOTHING in the tomcat
>>>>>> access
>>>>>> log.
>>>>>> 10.1.1.2 - - [22/Feb/2017:10:20:02 -0800] "-" 408 -
>>>>>>
>>>>>> So 408.  Timeout.  Hmm... Why?  We KNOW that it can connect from
http
>>>>>> ->
>>>>>> tomcat:18080 perfectly.   So is it a timeout BACK to the F5? That
i
>>>>>> dont
>>>>>> know yet.
>>>>>>
>>>>>
>>>>> It looks as if, in that case, httpd is trying to connect to tomcat, but
>>>>> - either it /is/ connecting, and sending the request, but tomcat never
>>>>> answers
>>>>> - or it is connecting to something else, which isn't tomcat
>>>>> (but which accepts the request and doesn't answer either)
>>>>>
>>>>> I am not *sure*, but I believe, that if httpd was trying to connect to
>>>>> something that isn't tomcat, and is not listening on port 18080, it would
>>>>> get a different error, and a different logfile message.  So it would
seem
>>>>> that at least the TCP connection works.
>>>>>
>>>>
>>>>
>>>> Is the F5 loadbalancer configured to make reverse proxy to a SSL http?
>>>> The directive SSLProxyEngine onthe F5 loadbalencer has to be enabled.
>>>> http://httpd.apache.org/docs/current/mod/mod_ssl.html#sslproxyengine
>>>>
>>>> If the loadbalancer it's delegating the SSL work to the server1
>>>> server1 https listen = 172.1.1.1:23270
>>>>
>>>> then the F5 could be connecting as http regular TCP connection to a SSL
>>>> TCP one and therefor the SSL handshake it's not happening.
>>>>
>>>> Antonio
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Now there is still a puzzle there : when you connect as
>>>>> browser --> httpd --> tomcat
>>>>> then everything works : httpd connects to tomcat, sends a request, gets
>>>>> a response, and returns it to the browser.
>>>>>
>>>>> But when you do the same as :
>>>>> browser --> F5 --> httpd --> tomcat
>>>>> it "does not work", and the problem seems to be at httpd --> tomat
>>>>> Why would it make a difference in httpd --> tomcat, if it is either
the
>>>>> browser or the F5 which sends the request to httpd ?
>>>>> the httpd --> tomcat connection should be the same, no matter what.
>>>>>
>>>>> Hmm, for now as puzzled as you are.
>>>>> Maybe we have a quantum phenomenon here, where the mere fact of you
>>>>> looking at the logfile, changes what happened beforehand. Just kidding..
>>>>>
>>>>> Informational note : the Apache access log line, is written when Apache
>>>>> *has finished* processing the request (including the possible round-trip
>>>>> through tomcat) and has sent the response to the client. (That's because,
>>>>> for example, it can also write the time spent, and the size of the
>>>>> response; so it has to know those before it can write the log line).
>>>>> I presume that it is the same for tomcat.
>>>>> Just mentioning this, so that you would maybe have a second look at the
>>>>> tomcat access log, after a few minutes, to make sure that there really
is
>>>>> no corresponding line there.
>>>>> It could be that tomcat actually /is/ receiving the request, and
>>>>> processing it, but that it takes a long time, and that httpd tires of
>>>>> waiting and already times out in the meantime. In such a case, you would
>>>>> probably see an error some time later in the tomcat error log, when it
is
>>>>> trying to eventually return the response, and finds a closed connection
>>>>> instead.
>>>>> Now again, why it would do that in one case and not the other, is still
>>>>> a mystery.
>>>>>
>>>>> Another note : in the "timeout" log line above, httpd is not showing
>>>>> the request URL which triggered this timeout.
>>>>> a) it may be showing some additional information in the Apache error
log
>>>>> b) these exists an httpd add-on module (mod_log_forensic ?) which could
>>>>> provide more details in the log, about what happens precisely
>>>>>
>>>>>
>>>>>
>>>>>> On Tue, Feb 21, 2017 at 3:39 PM, André Warnier (tomcat) <aw@ice-sa.com
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> On 21.02.2017 23:28, Aaron Gray wrote:
>>>>>>>
>>>>>>> Antonio:  The Tomcat server has no knowledge of the F5, or that
it is
>>>>>>>> being
>>>>>>>> fronted by an Apache HTTP Server.  I do SSL termination in
Apache
>>>>>>>> HTTP
>>>>>>>> Server, and clear-text from HTTP to Tomcat.
>>>>>>>> My redirect port for the normal HTTP listen in Tomcat is
commented
>>>>>>>> out.
>>>>>>>>        <Connector port="18080" protocol="HTTP/1.1"
>>>>>>>>                   connectionTimeout="20000" />
>>>>>>>>        <!-- A "Connector" using the shared thread pool-->
>>>>>>>>        <!--
>>>>>>>>        <Connector executor="tomcatThreadPool"
>>>>>>>>                   port="8080" protocol="HTTP/1.1"
>>>>>>>>                   connectionTimeout="20000"
>>>>>>>>                   redirectPort="8443" />
>>>>>>>>        -->
>>>>>>>>
>>>>>>>> Andre:
>>>>>>>> The URL I am using is https://loadbalancer.domain.com
>>>>>>>> It is listening on port 80 and 443, if you hit 80, internally
it
>>>>>>>> redirects
>>>>>>>> you to 443.  No SSL cert on the F5 load balancer.  It simply
sends
>>>>>>>> the
>>>>>>>> traffic to one of the two HTTP servers (round-robin, also
tried
>>>>>>>> persistence, no difference).  The HTTP server is listening
only for
>>>>>>>> HTTPS
>>>>>>>> on 23270/tcp.
>>>>>>>> When I access these DMZ webservers via the F5 load balancer
(to
>>>>>>>> which I
>>>>>>>> dont have access to, but the network folks configure for
me), it
>>>>>>>> hangs.
>>>>>>>> Eventually returns: server1 https listen = 172.1.1.1:23270
>>>>>>>>
>>>>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>>>>> cant load.
>>>>>>>>
>>>>>>>> No idea why the URL is being re-written with the ":23270".
>>>>>>>> I added static content to the server.xml on 10.1.1.1 (Tomcat)
to
>>>>>>>> test:
>>>>>>>> <Context docBase="/path/to/tomcat/static" path="/static"
/>
>>>>>>>> Then put a simple index.html in there.  Accessing via the
Apache Web
>>>>>>>> Servers works fine, but if you hit it with the Load Balancer
it once
>>>>>>>> again
>>>>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>>>>
>>>>>>>> Hitting https://loadbalancer.domain.com
>>>>>>>> I see my "Hello world!" which is all that is in index.html.
This is
>>>>>>>> the
>>>>>>>> DocumentRoot of HTTP, and *not* proxied over at this time.
>>>>>>>>
>>>>>>>>
>>>>>>> So in this case, there is no delay, and you get the Apache
>>>>>>> httpd-hosted
>>>>>>> "index.html" containing "Hello World. Right ?
>>>>>>>
>>>>>>>     Only
>>>>>>>
>>>>>>> /SelfService and /static are proxied
>>>>>>>> /static just being my test of static content, but still served
up by
>>>>>>>> Tomcat..
>>>>>>>>
>>>>>>>> It's exactly 30 seconds before the page cannot be loaded
when trying
>>>>>>>> anything proxied to Tomcat, but also accessed via the F5
load
>>>>>>>> balancer.
>>>>>>>> Not sure where the 30 seconds comes from; perhaps a load
balancer
>>>>>>>> time
>>>>>>>> out,
>>>>>>>> as I dont see a "30" in my httpd configurations or my tomcat
>>>>>>>> server.xml
>>>>>>>>
>>>>>>>>
>>>>>>>> You can certainly look at the Apache httpd logs, and the
tomcat
>>>>>>> logs, to
>>>>>>> see if you get a request or not.
>>>>>>> In Apache httpd, you can set the loglevel individually for mod_proxy
>>>>>>> (if
>>>>>>> you are running v 2.4), and it should show something if it gets
this
>>>>>>> request and forwards it to tomcat.
>>>>>>> In tomcat, you can either enable an access log (which will show
if it
>>>>>>> receives this request), or you could temporarily remove/rename
the
>>>>>>> /static
>>>>>>> webapp. This way, it should trigger an error "not found" which
you
>>>>>>> would
>>>>>>> also see in the error log.
>>>>>>>
>>>>>>> There should be nothing between them to hinder it.  We have many
load
>>>>>>>
>>>>>>>> balancers and this one specifically you dont need to open
any
>>>>>>>> firewall
>>>>>>>> requests for the specific networks the HTTP servers are on.
I did
>>>>>>>> have to
>>>>>>>> get the firewall opened up to allow me to hit
>>>>>>>> https://loadbalancer.domain.com because the VIP for "
>>>>>>>> loadbalancer.domain.com"
>>>>>>>> is in the DMZ, and my Desktop & VPN networks cannot hit
it on 80/443
>>>>>>>> without opening holes.  But beyond that, any connection from
the F5
>>>>>>>> to the
>>>>>>>> HTTP Server should be 100% open bi-directional, since same
subnet.
>>>>>>>>
>>>>>>>>
>>>>>>>> But something isn't working, otherwise you would not be asking.
>>>>>>> So,
>>>>>>>
>>>>>>> a) hitting the tomcat webapps through httpd seems to be working
fine
>>>>>>>     (browser -> httpd:23270 -> tomcat:18080 -> webapp
or static)
>>>>>>>
>>>>>>> b) hitting a non-proxied-to-tomcat resource of httpd seems to
work
>>>>>>> fine
>>>>>>> too, even through the F5
>>>>>>>     (browser -> F5:443 -> httpd:23270 -> html page)
>>>>>>>
>>>>>>> c) it is only when you do :
>>>>>>>     (browser -> F5:443 -> httpd:23270 -> tomcat:18080
-> webapp or
>>>>>>> static)
>>>>>>>     that you see this issue
>>>>>>>
>>>>>>> It would really help if you looked in the logs of both httpd
and
>>>>>>> tomcat,
>>>>>>> and checked for differences betweens cases a, b and c above.
>>>>>>>
>>>>>>> I believe that the F5 message with the port 23270 is a minor
issue, of
>>>>>>> information disclosure by the F5, that it should not disclose.
>>>>>>>
>>>>>>> But the reason why it returns this error is obviously that in
that
>>>>>>> case,
>>>>>>> it does not get a response from his request to httpd.
>>>>>>> The reason for this response not coming back to the F5 (in case
c
>>>>>>> only),
>>>>>>> can be due to either httpd or tomcat. But F5 doesn't know about
>>>>>>> tomcat. So
>>>>>>> for the F5, it is httpd which is not responding. Thus,
>>>>>>> - either httpd is never getting the request from the F5 (unlikely,
>>>>>>> because
>>>>>>> in b above it gets it and responds)
>>>>>>> - or httpd is getting the request from the F5, but not forwarding
it
>>>>>>> to
>>>>>>> tomcat, but also not returning an immediate error response to
F5
>>>>>>> (which
>>>>>>> seems also unlikely, because of a and b)
>>>>>>> - or httpd is getting the request, forwarding it to tomcat, but
not
>>>>>>> getting a response from tomcat. So
>>>>>>>     - either tomcat is never getting the request from httpd (but
in a,
>>>>>>> it
>>>>>>> gets it)
>>>>>>>     - or tomcat is getting the request from httpd, but not responding
>>>>>>> (but
>>>>>>> in a, it does)
>>>>>>>     - or tomcat is getting the request and responding, but the
response
>>>>>>> never gets back to httpd (but in a, it does)
>>>>>>>
>>>>>>> So if a and b and c are all accurate, there is something apparently
>>>>>>> illogical happening.
>>>>>>> This would lead to the conclusion that a and b and c cannot all
be
>>>>>>> accurate.
>>>>>>>
>>>>>>> The logs.. ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 21, 2017 at 2:05 PM, Antonio S. Cofino <cofinoa@gmail.com
>>>>>>>>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Aaron, on tomcat instances change the redirectPort attributte
on the
>>>>>>>> http
>>>>>>>>
>>>>>>>>> conectó to the loabbalancer's port 443
>>>>>>>>>
>>>>>>>>> My guess is that your webapp has restriction rule requesting
SSL con
>>>>>>>>> fidntial channel. Therefore the non-confidential to the
18080 port
>>>>>>>>> from
>>>>>>>>> the
>>>>>>>>> balancer are redirected to the 23270 port, but it should
be 443.
>>>>>>>>>
>>>>>>>>> Antonio
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El 21/2/2017 19:46, "Aaron Gray" <aaronmgray@gmail.com>
escribió:
>>>>>>>>>
>>>>>>>>> I have an application server from a vendor that comes
bundled with
>>>>>>>>> an
>>>>>>>>> additional Apache Tomcat server.  The webapp SelfService.war
is
>>>>>>>>> vendor
>>>>>>>>> supplied too.
>>>>>>>>>
>>>>>>>>> Here's my problem (IP's replaced to protect the innocent):
>>>>>>>>>
>>>>>>>>> networks:
>>>>>>>>> DMZ=172.x.x.x
>>>>>>>>> INTERNAL=10.x.x.x
>>>>>>>>>
>>>>>>>>> server1 https listen = 172.1.1.1:23270
>>>>>>>>> server2 https listen = 172.1.1.2:23270
>>>>>>>>> F5 load balancer hostname = loadbalancer.domain.com:443
>>>>>>>>> backend tomcat server = 10.1.1.1:18080
>>>>>>>>>
>>>>>>>>> mod_proxy configuration:
>>>>>>>>> ProxyPass /SelfService http://10.1.1.1:18080/SelfService
>>>>>>>>> ProxyPassReverse /SelfService http://10.1.1.1:18080/SelfService
>>>>>>>>>
>>>>>>>>> When I access these DMZ webservers which mod_proxy back
to Apache
>>>>>>>>> Tomcat
>>>>>>>>> as:
>>>>>>>>> https://172.1.1.1:23270/SelfService
>>>>>>>>> and
>>>>>>>>> https://172.1.1.2:23270/SelfService <https://172.1.1.1:23270/SelfS
>>>>>>>>> ervice
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> They load properly. Perfectly, every time!
>>>>>>>>>
>>>>>>>>> When I access these DMZ webservers via the F5 load balancer
(to
>>>>>>>>> which I
>>>>>>>>> dont have access to, but the network folks configure
for me), it
>>>>>>>>> hangs.
>>>>>>>>> Eventually returns:
>>>>>>>>> https://loadbalancer.domain.com:23270/SelfService
>>>>>>>>> cant load.
>>>>>>>>>
>>>>>>>>> No idea why the URL is being re-written with the ":23270".
>>>>>>>>> I added static content to the server.xml on 10.1.1.1
(Tomcat) to
>>>>>>>>> test:
>>>>>>>>> <Context docBase="/path/to/tomcat/static" path="/static"
/>
>>>>>>>>> Then put a simple index.html in there.  Accessing via
the Apache Web
>>>>>>>>> Servers works fine, but if you hit it with the Load Balancer
it once
>>>>>>>>> again
>>>>>>>>> adds the https://loadbalancer.domain.com:23270/static
>>>>>>>>>
>>>>>>>>> Do you have any thoughts?  Thanks so much, I have been
working with
>>>>>>>>> this
>>>>>>>>> for weeks now with no success
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message