httpd-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <slawomir.jano...@postfinance.ch>
Subject RE: Problem with DNS lookup caching in reverse proxy
Date Fri, 13 May 2011 09:19:28 GMT
Hi everybody,

I'm happy to announce that I was now able to find a solution to this problem which seems to
work for me.
I would like to discuss my solution with you experts for possible corrections and/or improvements.

1. As I didn't find any place where I could put an arbitrarily executed timer, I decided to
go with a Boolean option to enable/disable the DNS address issue resolving.

2. I have put the handling code into the function ap_proxy_release_connection() of module
proxy_util.c

3. The code makes a call to apr_sockaddr_info_get() to check if the destination address has
changed. If it has changed, the change is propagated to all places I thought to be relevant
and then the connection in question is marked as being due to be closed. This is the code
I came up with:

if (dnsto) {
	apr_status_t return_value;
	const char *hostname = conn->worker->hostname;
	const apr_port_t port = conn->worker->port;
	apr_sockaddr_t *old_sock_addr = conn->addr;
	apr_sockaddr_t *new_sock_addr;
	apr_pool_t *pool;
	return_value = apr_pool_create(&pool, conn->pool);
	/*
	 * Obtain the currently valid address from the OS
	 */
	apr_sockaddr_info_get(&new_sock_addr, hostname, APR_UNSPEC, port, 0, pool);
	get_addr(old_address, old_sock_addr);
	get_addr(new_address, new_sock_addr);
	ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
			"proxy: ip of connection is %pI, the current resolved ip is %pI",
			old_sock_addr, new_sock_addr);
	if (strcmp(old_address, new_address)) {
		ap_log_error(APLOG_MARK, APLOG_INFO, 0, s,
			"proxy: Invalidating the connection due to change in dns resolved address.");
		/*
		 * If the address has changed, store the changed
		 * address in all relevant places and mark the
		 * connection as due to be closed.
		 */
		conn->worker->cp->addr = new_sock_addr;
		conn->addr = new_sock_addr;
		conn->connection->remote_addr = new_sock_addr;
		conn->close = 1;
	}
}

This code is located right before the call to connection_cleanup()
The get_addr() function isn't really necessary and is used here just for debugging purposes.
The comparison (at the moment a strcmp()) could easily be replaced by a check based on the
equality of the two apr_sockaddr_t-s.

The code as presented here has been tested on Linux. I hope, although I didn't test it yet,
that it also will work on Solaris, as this is my real target platform. With this hack, as
soon as the DNS resolver switches the ip address of the target, all the connections in the
proxy being released are checked and marked to be closed. They are rebuilt at the time of
the next access with the new address. This results in a correct switch-over. Backed up with
the already possible smax=0 and ttl=30 settings I have a double-strategy where all active
connections are closed more or less immediately (as soon as they are released), while all
less active connections are "garbage collected" by the normal connection inactivity timeout.
This is exactly what I want to achieve.

There are still some questions, though:
A. Is this the right place to put such a check?
B. Is it possible to have a timer to do this check instead?
C. Is the pool handling correct or does it lead to a memory leak?
D. Is the method to clean up and close the connections with obsolete ip addresses (the last
four lines of the code above) the right approach?

Thanks for any help.
Regards
Slawo.

-----Ursprüngliche Nachricht-----
Von: Janotta Slawomir Richard, PF54 extern 
Gesendet: Montag, 2. Mai 2011 11:26
An: i.galic@brainsware.org; dev@httpd.apache.org
Betreff: AW: Problem with DNS lookup caching in reverse proxy

Hi Igor and everybody :-)
Unfortunately, this is exactly what won't work for me.
The point is: In our case the remote address is still valid. The "old" node is not to be shut
down so the connection to it is technically still valid. Instead, the full qualified address
if looked up via the resolver now points to a "new" node. If I'm correct, in this case there
won't be any error. And if there is a lot of traffic on the connection, there also won't be
any timeout and the connection will be happy accessing the "old" node because it simply uses
the stored ip-address and doesn't see any reason to discard it. If I set ttl to say 1 second
(way too small for me in fact) and the connection is accessed every 100ms, there is no way
to timeout the connection. The only solution I can think of is to force the expiry after a
set timeout.
Having said that I would like to ask some additional help with implementing such a solution.
Unfortunately, I'm pretty new to Apache and have to work it out "the hard way". So far I have
worked out how to enable the additional parameter for the mod_proxy. I have done it by adding
the setting dnsto (for DNS timeout) to the structure proxy_worker in mod_proxy.h. The question
is: where can I register the timeout to be used and how do I force the worker to re-resolve
the address.
Regards
Slawo.

-----Ursprüngliche Nachricht-----
Von: Igor Galić [mailto:i.galic@brainsware.org] 
Gesendet: Dienstag, 26. April 2011 23:45
An: dev@httpd.apache.org
Betreff: Re: Problem with DNS lookup caching in reverse proxy



----- Original Message -----
> Hello everybody,
> 
> I have tried to solve my issue by contacting the users@ mailing list
> but meanwhile I think this is the more appropriate list to address
> it.
> I have issues with the mod_proxy resolving a full qualified
> servername via DNS and caching it in its connection pool so a change
> in the resolver doesn't get noticed by mod_proxy. After trying all
> the different possibilities that came to mind I'm back at where it
> all started.
> I still cannot find a solution to the problem that mod_proxy caches
> the IP address resolved by DNS and thus doesn't acknowledge all
> changes to it. With the help of Igor Galić on the users@ mailing
> list and by means of Bug Report 50551 I am able to get about 95% of
> the cases solved. But there still are cases where the resolved
> address won't be changed.
> But first let me describe the configuration more deeply:
> 
> What we have is a configuration where the Apache Reverse Proxy (RPX)
> is used to allow transparent switch of the backend JEE server. We
> solve this by configuring an alias to the service on the backend.
> The alias is then entered in whatever DNS resolver we use to point
> to the one system which is currently meant to receive the request.
> This allows us to make application changes on one system while the
> other is still working, then transparently switching to the updates
> server to do maintenance on the other.
> 
> Now the problem with this is that Apache, or more exactly the
> mod_proxy, has a connection pool mechanism which apparently binds to
> a once resolved IP address "forever". See the type struct
> proxy_conn_pool in file mod_proxy.h or just my description in Bug
> Report 50551.
> 
> It appears to me, and I believe I was able to show this by extensive
> testing, that the member variable apr_sockaddr_t *addr /* Preparsed
> remote address info */
> in this struct is responsible for this behavior and that this
> preparsed remote address is invalidated only if the last pooled
> connection in this connection pool is timed out.
> 
> In my opinion the best solution to this problem would be to add an
> additional perameter to mod_proxy to force a renewal of the DNS
> resolve procedure for this address after a set timeout like it is
> possible in mod_weblogic. Unfortunately this is way beyond my

Instead of re-resolving after a set timeout, it would make sense
to re-resolve after a timeout occured, marking the worker in error.

The only question is: Is that the struct shared across the pool?

> capabilities at the moment to implement it myself. But I wonder why
> there seems to be no need for such an option so nobody implemented
> it yet. Perhaps there is another I'm still not aware of?
> 
> Regards
> Slawo

Sicherheitshinweis:
Dieses E-Mail von PostFinance ist signiert. Weitere Informationen finden Sie unter: 
https://www.postfinance.ch/e-signature.
Geben Sie Ihre Sicherheitselemente niemals Dritten bekannt.
Mime
View raw message