cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ilya <ilya.mailing.li...@gmail.com>
Subject Re: Load Balancer (HAProxy) - TPROXY passthrough
Date Wed, 01 Apr 2015 19:27:35 GMT
Logan,

My 2cents,

While many including myself dont qualify as developers on this list, 
there are some development parts anyone can do - without knowing Java.

Specifically, agentless config management does not require much or any 
java programing in first stage. Perhaps the portion where a config file 
for vRouter needs to be fetched and apply by dfault - needs some 
enhancement, everything else can be handled by savy system person.

With that said, it should be trivial to add agent-less config management 
to router vm without any hardcore java expertise.

Regards
ilya

On 4/1/15 10:13 AM, Logan Barfield wrote:
> Hi Ilya,
>
> I definitely like that solution, and there is discussion elsewhere on
> overhauling how VRs are configured.
>
> Unfortunately I am not a developer.  Any changes I make are generally
> to existing code, and are based off of context and testing more than
> anything else.  I don't currently have the time or knowledge required
> to implement such a big change myself, and even if I did I wouldn't
> know where to start.  I would be more than happy to discuss something
> like that with other community members, and offer patches/testing
> where I can.  I don't see myself being able to take the lead on such a
> task though.
>
> For now I'd like to address this particular issue (transparent proxy
> support), to see if the community has any objections or concerns about
> the proposed changes.  We can (and probably will) end up just creating
> our own internal branch, but I'd rather be able to commit any changes
> we make to the project for ease of upgrades, support, etc.
>
> Thank You,
>
> Logan Barfield
> Tranquil Hosting
>
>
> On Wed, Apr 1, 2015 at 12:58 PM, ilya <ilya.mailing.lists@gmail.com> wrote:
>> Logan
>>
>> For sometime now, idea was kicked around of extending VR services where it
>> can take custom configurations.
>>
>> In theory, VR should be driven via config management tool (i.e. puppet,
>> salt, chef, ansible - or whatever works).
>>
>> Instead of hacking this in portions, perhaps you can spend sometime
>> analizing how VR is build and how you we can extend the functionality via
>> external config management.
>>
>> For example, when vrouter comes up, because we already have ssh keys dropped
>> on each VR, just on this premise alone - we can do just about anything. I'm
>> thinking salt or ansible would be a better solution as it is SSH driven and
>> agentless.
>>
>> Try porting your changes into salt or ansible (or anything else agentless)
>> and propose a framework on how we can enhance this.
>>
>> I'm thinking, for each Account (as it gets a VR), we need to have a ansible
>> config file that will be pushed through once VR comes up to configure
>> remaining custom bits/components to your liking.
>>
>> Thoughts?
>>
>> regards
>> ilya
>>
>>
>> On 4/1/15 9:34 AM, Logan Barfield wrote:
>>> We've been running into some issues with the Advanced Zone/Isolated
>>> Network Load Balancer, and in working through them we've come up with
>>> some ideas for how the functionality can be improved.
>>>
>>> The first issue we hit was with HTTP load balancing.  We had a site
>>> that was sending back larger than average HTTP headers with 302
>>> redirects.  This was causing HAProxy to return 502 errors to clients.
>>>
>>> This is apparently a known issue with HAProxy when using the default
>>> "tune.bufsize" and "tune.maxrewrite" settings.The official HAProxy
>>> documentation recommends changing these from the defaults.  We were
>>> able to work around the problem by manually setting "tune.maxrewrite
>>> 1024" in the haproxy.cfg on the virtual router.  This resolved most of
>>> the 502 errors, and would have probably resolved all of them with more
>>> tuning.  The problem is that this change obviously wouldn't survive
>>> upgrades or VR rebuilds.
>>>
>>> To fix the problem on a more permanent basis we changed the
>>> KeepAliveEnabled Network Offering setting introduced a few versions
>>> ago.  This directs HAProxy to use TCP mode instead of HTTP mode for
>>> rules configured on port 80.  This solution works for the most part,
>>> but there are a couple of problems:
>>> 1) There doesn't appear to be support for this setting in the UI.
>>> That's understandable as the UI is way behind the current feature set.
>>> 2) There doesn't appear to be support for this setting in the API,
>>> either when creating or updating Network Offerings.  This is a bit of
>>> a problem.  We had to make the change in the database directly, which
>>> is very dirty.
>>> 3) TCP connections don't grab HTTP headers, so they can't send the
>>> real IP to HTTP/Nginx in the X-Forwarded-For header.
>>>
>>> To fix these issues I suggest the following changes be made:
>>> 1) Add the "KeepAliveEnabled" option to the Network Offering API
>>> commands.  I really have no idea how to do this, so I'll try to flag
>>> the original committer to see if they can do so.
>>>
>>> 2) Add a new option for TPROXY support.  The current VR kernel and
>>> HAProxy version have TPROXY support built in, so having the option (on
>>> a per LB rule basis) would be great.  This would allow for using TCP
>>> mode in HAProxy, while still passing the real IP through to the
>>> backend services.  To accomplish this I would suggest adding the
>>> necessary IPtables rules to the VR either by default, or when Load
>>> Balancing is first enabled.  Then a flag can be added to the create LB
>>> rule command to either enable or disable the transparent proxy
>>> setting.
>>> The necessary IPtables rules are:
>>>
>>> iptables -t mangle -N DIVERT
>>> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
>>> iptables -t mangle -A DIVERT -j MARK --set-mark 111
>>> iptables -t mangle -A DIVERT -j ACCEPT
>>> ip rule add fwmark 111 lookup 100
>>> ip route add local 0.0.0.0/0 dev lo table 100
>>>
>>> Then for each LB rule created with the "transparent" option enabled,
>>> add the following setting to the rule configuration:
>>>
>>> source 0.0.0.0 usesrc clientip
>>>
>>> The last configuration change is to remove the user/group or uid/gid
>>> options in the haproxy config, otherwise HAProxy won't start with
>>> "usesrc" enabled.
>>>
>>> It is also recommended to enable
>>> /proc/sys/net/ipv4/conf/eth0/send_redirects, but I haven't noticed any
>>> issues with it disabled either, and I don't know what else it might
>>> affect.
>>>
>>> 3) The last suggestion would be to move away from hard coding
>>> configuration directives for VR services (like HAProxy:
>>>
>>> https://github.com/apache/cloudstack/blob/5091d0f5c5b03cb8658f2d974103261341080825/core/src/com/cloud/network/HAProxyConfigurator.java)
>>>    Doing this makes implementing changes a hassle since it involves
>>> rebuilding/upgrading CloudStack to accomplish anything, even small
>>> edits.  For a production environment this is ill advised if not
>>> impossible.  In general it would make sense to make persistent changes
>>> to VR services possible without recompiling code or rebooting the VRs.
>>> I believe that's part of a bigger issue though, as I've seen some
>>> discussion about it on the list.
>>>
>>>
>>> If anyone actually makes it through this, I'd appreciate any feedback
>>> on things I may not be considering, or reasons not to implement these
>>> changes.  I doubt I'll get enough traction for an actual developer to
>>> help, so I'll probably end up hacking these in myself and committing
>>> them.  I just wanted to see what the community thought first.
>>>
>>>
>>> Thank You,
>>>
>>> Logan Barfield
>>> Tranquil Hosting
>>


Mime
View raw message