cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ilya <>
Subject Re: Load Balancer (HAProxy) - TPROXY passthrough
Date Wed, 01 Apr 2015 19:27:35 GMT

My 2cents,

While many including myself dont qualify as developers on this list, 
there are some development parts anyone can do - without knowing Java.

Specifically, agentless config management does not require much or any 
java programing in first stage. Perhaps the portion where a config file 
for vRouter needs to be fetched and apply by dfault - needs some 
enhancement, everything else can be handled by savy system person.

With that said, it should be trivial to add agent-less config management 
to router vm without any hardcore java expertise.


On 4/1/15 10:13 AM, Logan Barfield wrote:
> Hi Ilya,
> I definitely like that solution, and there is discussion elsewhere on
> overhauling how VRs are configured.
> Unfortunately I am not a developer.  Any changes I make are generally
> to existing code, and are based off of context and testing more than
> anything else.  I don't currently have the time or knowledge required
> to implement such a big change myself, and even if I did I wouldn't
> know where to start.  I would be more than happy to discuss something
> like that with other community members, and offer patches/testing
> where I can.  I don't see myself being able to take the lead on such a
> task though.
> For now I'd like to address this particular issue (transparent proxy
> support), to see if the community has any objections or concerns about
> the proposed changes.  We can (and probably will) end up just creating
> our own internal branch, but I'd rather be able to commit any changes
> we make to the project for ease of upgrades, support, etc.
> Thank You,
> Logan Barfield
> Tranquil Hosting
> On Wed, Apr 1, 2015 at 12:58 PM, ilya <> wrote:
>> Logan
>> For sometime now, idea was kicked around of extending VR services where it
>> can take custom configurations.
>> In theory, VR should be driven via config management tool (i.e. puppet,
>> salt, chef, ansible - or whatever works).
>> Instead of hacking this in portions, perhaps you can spend sometime
>> analizing how VR is build and how you we can extend the functionality via
>> external config management.
>> For example, when vrouter comes up, because we already have ssh keys dropped
>> on each VR, just on this premise alone - we can do just about anything. I'm
>> thinking salt or ansible would be a better solution as it is SSH driven and
>> agentless.
>> Try porting your changes into salt or ansible (or anything else agentless)
>> and propose a framework on how we can enhance this.
>> I'm thinking, for each Account (as it gets a VR), we need to have a ansible
>> config file that will be pushed through once VR comes up to configure
>> remaining custom bits/components to your liking.
>> Thoughts?
>> regards
>> ilya
>> On 4/1/15 9:34 AM, Logan Barfield wrote:
>>> We've been running into some issues with the Advanced Zone/Isolated
>>> Network Load Balancer, and in working through them we've come up with
>>> some ideas for how the functionality can be improved.
>>> The first issue we hit was with HTTP load balancing.  We had a site
>>> that was sending back larger than average HTTP headers with 302
>>> redirects.  This was causing HAProxy to return 502 errors to clients.
>>> This is apparently a known issue with HAProxy when using the default
>>> "tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy
>>> documentation recommends changing these from the defaults.  We were
>>> able to work around the problem by manually setting "tune.maxrewrite
>>> 1024" in the haproxy.cfg on the virtual router.  This resolved most of
>>> the 502 errors, and would have probably resolved all of them with more
>>> tuning.  The problem is that this change obviously wouldn't survive
>>> upgrades or VR rebuilds.
>>> To fix the problem on a more permanent basis we changed the
>>> KeepAliveEnabled Network Offering setting introduced a few versions
>>> ago.  This directs HAProxy to use TCP mode instead of HTTP mode for
>>> rules configured on port 80.  This solution works for the most part,
>>> but there are a couple of problems:
>>> 1) There doesn't appear to be support for this setting in the UI.
>>> That's understandable as the UI is way behind the current feature set.
>>> 2) There doesn't appear to be support for this setting in the API,
>>> either when creating or updating Network Offerings.  This is a bit of
>>> a problem.  We had to make the change in the database directly, which
>>> is very dirty.
>>> 3) TCP connections don't grab HTTP headers, so they can't send the
>>> real IP to HTTP/Nginx in the X-Forwarded-For header.
>>> To fix these issues I suggest the following changes be made:
>>> 1) Add the "KeepAliveEnabled" option to the Network Offering API
>>> commands.  I really have no idea how to do this, so I'll try to flag
>>> the original committer to see if they can do so.
>>> 2) Add a new option for TPROXY support.  The current VR kernel and
>>> HAProxy version have TPROXY support built in, so having the option (on
>>> a per LB rule basis) would be great.  This would allow for using TCP
>>> mode in HAProxy, while still passing the real IP through to the
>>> backend services.  To accomplish this I would suggest adding the
>>> necessary IPtables rules to the VR either by default, or when Load
>>> Balancing is first enabled.  Then a flag can be added to the create LB
>>> rule command to either enable or disable the transparent proxy
>>> setting.
>>> The necessary IPtables rules are:
>>> iptables -t mangle -N DIVERT
>>> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
>>> iptables -t mangle -A DIVERT -j MARK --set-mark 111
>>> iptables -t mangle -A DIVERT -j ACCEPT
>>> ip rule add fwmark 111 lookup 100
>>> ip route add local dev lo table 100
>>> Then for each LB rule created with the "transparent" option enabled,
>>> add the following setting to the rule configuration:
>>> source usesrc clientip
>>> The last configuration change is to remove the user/group or uid/gid
>>> options in the haproxy config, otherwise HAProxy won't start with
>>> "usesrc" enabled.
>>> It is also recommended to enable
>>> /proc/sys/net/ipv4/conf/eth0/send_redirects, but I haven't noticed any
>>> issues with it disabled either, and I don't know what else it might
>>> affect.
>>> 3) The last suggestion would be to move away from hard coding
>>> configuration directives for VR services (like HAProxy:
>>>    Doing this makes implementing changes a hassle since it involves
>>> rebuilding/upgrading CloudStack to accomplish anything, even small
>>> edits.  For a production environment this is ill advised if not
>>> impossible.  In general it would make sense to make persistent changes
>>> to VR services possible without recompiling code or rebooting the VRs.
>>> I believe that's part of a bigger issue though, as I've seen some
>>> discussion about it on the list.
>>> If anyone actually makes it through this, I'd appreciate any feedback
>>> on things I may not be considering, or reasons not to implement these
>>> changes.  I doubt I'll get enough traction for an actual developer to
>>> help, so I'll probably end up hacking these in myself and committing
>>> them.  I just wanted to see what the community thought first.
>>> Thank You,
>>> Logan Barfield
>>> Tranquil Hosting

View raw message