cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ilya <ilya.mailing.li...@gmail.com>
Subject Re: Load Balancer (HAProxy) - TPROXY passthrough
Date Wed, 01 Apr 2015 16:58:36 GMT
Logan

For sometime now, idea was kicked around of extending VR services where 
it can take custom configurations.

In theory, VR should be driven via config management tool (i.e. puppet, 
salt, chef, ansible - or whatever works).

Instead of hacking this in portions, perhaps you can spend sometime 
analizing how VR is build and how you we can extend the functionality 
via external config management.

For example, when vrouter comes up, because we already have ssh keys 
dropped on each VR, just on this premise alone - we can do just about 
anything. I'm thinking salt or ansible would be a better solution as it 
is SSH driven and agentless.

Try porting your changes into salt or ansible (or anything else 
agentless) and propose a framework on how we can enhance this.

I'm thinking, for each Account (as it gets a VR), we need to have a 
ansible config file that will be pushed through once VR comes up to 
configure remaining custom bits/components to your liking.

Thoughts?

regards
ilya

On 4/1/15 9:34 AM, Logan Barfield wrote:
> We've been running into some issues with the Advanced Zone/Isolated
> Network Load Balancer, and in working through them we've come up with
> some ideas for how the functionality can be improved.
>
> The first issue we hit was with HTTP load balancing.  We had a site
> that was sending back larger than average HTTP headers with 302
> redirects.  This was causing HAProxy to return 502 errors to clients.
>
> This is apparently a known issue with HAProxy when using the default
> "tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy
> documentation recommends changing these from the defaults.  We were
> able to work around the problem by manually setting "tune.maxrewrite
> 1024" in the haproxy.cfg on the virtual router.  This resolved most of
> the 502 errors, and would have probably resolved all of them with more
> tuning.  The problem is that this change obviously wouldn't survive
> upgrades or VR rebuilds.
>
> To fix the problem on a more permanent basis we changed the
> KeepAliveEnabled Network Offering setting introduced a few versions
> ago.  This directs HAProxy to use TCP mode instead of HTTP mode for
> rules configured on port 80.  This solution works for the most part,
> but there are a couple of problems:
> 1) There doesn't appear to be support for this setting in the UI.
> That's understandable as the UI is way behind the current feature set.
> 2) There doesn't appear to be support for this setting in the API,
> either when creating or updating Network Offerings.  This is a bit of
> a problem.  We had to make the change in the database directly, which
> is very dirty.
> 3) TCP connections don't grab HTTP headers, so they can't send the
> real IP to HTTP/Nginx in the X-Forwarded-For header.
>
> To fix these issues I suggest the following changes be made:
> 1) Add the "KeepAliveEnabled" option to the Network Offering API
> commands.  I really have no idea how to do this, so I'll try to flag
> the original committer to see if they can do so.
>
> 2) Add a new option for TPROXY support.  The current VR kernel and
> HAProxy version have TPROXY support built in, so having the option (on
> a per LB rule basis) would be great.  This would allow for using TCP
> mode in HAProxy, while still passing the real IP through to the
> backend services.  To accomplish this I would suggest adding the
> necessary IPtables rules to the VR either by default, or when Load
> Balancing is first enabled.  Then a flag can be added to the create LB
> rule command to either enable or disable the transparent proxy
> setting.
> The necessary IPtables rules are:
>
> iptables -t mangle -N DIVERT
> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
> iptables -t mangle -A DIVERT -j MARK --set-mark 111
> iptables -t mangle -A DIVERT -j ACCEPT
> ip rule add fwmark 111 lookup 100
> ip route add local 0.0.0.0/0 dev lo table 100
>
> Then for each LB rule created with the "transparent" option enabled,
> add the following setting to the rule configuration:
>
> source 0.0.0.0 usesrc clientip
>
> The last configuration change is to remove the user/group or uid/gid
> options in the haproxy config, otherwise HAProxy won't start with
> "usesrc" enabled.
>
> It is also recommended to enable
> /proc/sys/net/ipv4/conf/eth0/send_redirects, but I haven't noticed any
> issues with it disabled either, and I don't know what else it might
> affect.
>
> 3) The last suggestion would be to move away from hard coding
> configuration directives for VR services (like HAProxy:
> https://github.com/apache/cloudstack/blob/5091d0f5c5b03cb8658f2d974103261341080825/core/src/com/cloud/network/HAProxyConfigurator.java)
>   Doing this makes implementing changes a hassle since it involves
> rebuilding/upgrading CloudStack to accomplish anything, even small
> edits.  For a production environment this is ill advised if not
> impossible.  In general it would make sense to make persistent changes
> to VR services possible without recompiling code or rebooting the VRs.
> I believe that's part of a bigger issue though, as I've seen some
> discussion about it on the list.
>
>
> If anyone actually makes it through this, I'd appreciate any feedback
> on things I may not be considering, or reasons not to implement these
> changes.  I doubt I'll get enough traction for an actual developer to
> help, so I'll probably end up hacking these in myself and committing
> them.  I just wanted to see what the community thought first.
>
>
> Thank You,
>
> Logan Barfield
> Tranquil Hosting


Mime
View raw message