cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nux! <...@li.nux.ro>
Subject Re: HTTPS LB and x-forwarded-for
Date Fri, 17 Nov 2017 16:39:49 GMT
lol.... good find..

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Pierre-Luc Dion" <pdion@cloudops.com>
> To: "Wido den Hollander" <wido@widodh.nl>
> Cc: "dev" <dev@cloudstack.apache.org>, "users" <users@cloudstack.apache.org>
> Sent: Friday, 17 November, 2017 13:56:11
> Subject: Re: HTTPS LB and x-forwarded-for

> That's funny!
> 
> Cloudstack ui does not provide lb protocol options, but the api does and
> cloudstack already support proxy proto v1!!!
> 
> So that's cool!
> 
> Le 13 nov. 2017 09 h 18, "Wido den Hollander" <wido@widodh.nl> a écrit :
> 
>>
>> > Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pdion@cloudops.com
>> >:
>> >
>> >
>> > Hi,
>> >
>> > This is looking quite promising, I have a colleague that tested that last
>> > Friday, look like the proxy proto v1 work out of the box with Nginx, but
>> > would need an extra package for Apache 2.4 ?
>>
>> It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/
>> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>>
>> It's there upstream, but not in all packages.
>>
>> It can from this module:
>>
>> - https://github.com/roadrunner2/mod-proxy-protocol
>> - https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html
>>
>> They donated the code to go upstream and went into mod_remoteip but landed
>> in 2.4.28
>>
>> It will probably make it into Ubuntu 18.04 and CentOS 7.4.
>>
>> Wido
>>
>> > Otherwise, it seems to be a good way to do  https LB without complicated
>> > configuration and huge changes in CloudStack.
>> >
>> >
>> >
>> > On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nux@li.nux.ro> wrote:
>> >
>> > > Pierre-Luc,
>> > >
>> > > Haproxy docs say it should work for any kind of traffic as long as both
>> > > ends are PROXY-aware and it look like a majority of software is.
>> > > So, in short, yes.
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> > >
>> > > ----- Original Message -----
>> > > > From: "Pierre-Luc Dion" <pdion@cloudops.com>
>> > > > To: "Wido den Hollander" <wido@widodh.nl>
>> > > > Cc: "dev" <dev@cloudstack.apache.org>, "Khosrow Moossavi" <
>> > > kmoossavi@cloudops.com>, "Will Stevens"
>> > > > <wstevens@cloudops.com>, "Nux!" <nux@li.nux.ro>, "users"
<
>> > > users@cloudstack.apache.org>
>> > > > Sent: Friday, 10 November, 2017 15:32:38
>> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > >
>> > > > Hi Wido, do you know if this would work for https traffic too?
>> > > >
>> > > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wido@widodh.nl>
a
>> écrit :
>> > > >
>> > > >>
>> > > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
>> > > pdion@cloudops.com
>> > > >> >:
>> > > >> >
>> > > >> >
>> > > >> > I kind of like the proxy backend type, ill check on our end
if
>> that
>> > > would
>> > > >> > work but definitely a simple and efficient approach!
>> > > >> >
>> > > >>
>> > > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>> > > >>
>> > > >> Apache HTTPd supports PROXY since 2.4.28:
>> > > https://httpd.apache.org/docs/
>> > > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>> > > >>
>> > > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>> > > >>
>> > > >> Wido
>> > > >>
>> > > >> >
>> > > >> >
>> > > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wido@widodh.nl>
a
>> > > écrit :
>> > > >> >
>> > > >> > >
>> > > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nux@li.nux.ro>:
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > Wido,
>> > > >> > > >
>> > > >> > > > Excellent suggestion with the "transparent proxy",
I was not
>> > > aware of
>> > > >> > > that.
>> > > >> > > > I think that would be a great idea and wouldn't
require too
>> many
>> > > >> > > modifications, especially as Haproxy comes already with
the VR.
>> > > >> > > >
>> > > >> > >
>> > > >> > > It's indeed just a matter of a HAProxy config setting.
We could
>> > > make it
>> > > >> > > configurable per backend in HAProxy. Regular HTTP, TCP
or PROXY
>> for
>> > > >> example.
>> > > >> > >
>> > > >> > > That way your problem would be solved.
>> > > >> > >
>> > > >> > > Wido
>> > > >> > >
>> > > >> > > > To Paul:
>> > > >> > > > - imho the LB solution ACS ships now is a bit handicaped
since
>> > > you do
>> > > >> > > not know the remote host ip. You're flying blind unless
you use
>> > > google
>> > > >> > > analytics (and these things have gotten more and more
>> aggressively
>> > > >> filtered
>> > > >> > > by adblocks).
>> > > >> > > > Enhancing Haproxy as Wido suggested would go a
long way, it
>> > > wouldn't
>> > > >> > > break existing functionality and would also keep SSL
processing
>> off
>> > > >> the VR.
>> > > >> > > >
>> > > >> > > > --
>> > > >> > > > Sent from the Delta quadrant using Borg technology!
>> > > >> > > >
>> > > >> > > > Nux!
>> > > >> > > > www.nux.ro
>> > > >> > > >
>> > > >> > > > ----- Original Message -----
>> > > >> > > > > From: "Andrija Panic" <andrija.panic@gmail.com>
>> > > >> > > > > To: "users" <users@cloudstack.apache.org>
>> > > >> > > > > Cc: "Khosrow Moossavi" <kmoossavi@cloudops.com>,
"Will
>> > > Stevens" <
>> > > >> > > wstevens@cloudops.com>, "dev"
>> > > >> > > > > <dev@cloudstack.apache.org>, "Pierre-Luc
Dion" <
>> > > pdion@cloudops.com
>> > > >> >
>> > > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
>> > > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >> > > >
>> > > >> > > > > Wido,
>> > > >> > > > >
>> > > >> > > > > backend servers are not Linux only, for example
we have a
>> ton of
>> > > >> > > Windows
>> > > >> > > > > customers, some WEB solutions / IIS etc...
>> > > >> > > > >
>> > > >> > > > > @all - If we try to please/solve everyone's
proxying
>> > > >> > > solution/requirement -
>> > > >> > > > > this is impossible IMHO - I'm thinking more
about some "do
>> it as
>> > > >> you
>> > > >> > > like"
>> > > >> > > > > solution, to let customer write his own haproxy
config and
>> > > upoad it
>> > > >> > > (for
>> > > >> > > > > example, or something better?).
>> > > >> > > > >
>> > > >> > > > > We can support newer version of haproxy (1.5+)
which also
>> > > implement
>> > > >> > > > > "transarent proxy" (integrate with kernel
so to speak)  to
>> allow
>> > > >> > > TCP-level
>> > > >> > > > > connections to backend (TCP mode, not HTTP
mode) but to
>> still
>> > > >> > > "preserve"
>> > > >> > > > > remote IP by faking it (fake soruce IP = transarent
proxy).
>> > > >> > > > >
>> > > >> > > > > For the rest of configuration options,  I
would leave it to
>> the
>> > > >> > > customer
>> > > >> > > > > how he/she wants to configure rest of haproxy
configuration,
>> > > >> inlcuding
>> > > >> > > > > custom checks, etc. Haproxy configuration
is never-ending
>> story,
>> > > >> and we
>> > > >> > > > > probably should allow custom sripts/configuration
instead of
>> > > >> trying to
>> > > >> > > > > provide GUI/API way to configure everything
(which is
>> > > >> impossible...)
>> > > >> > > > >
>> > > >> > > > > Just my 2 cents...
>> > > >> > > > >
>> > > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander
<
>> wido@widodh.nl
>> > > >
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > >>
>> > > >> > > > >> > Op 8 november 2017 om 14:59 schreef
Pierre-Luc Dion <
>> > > >> > > pdion@cloudops.com
>> > > >> > > > >> >:
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > Same challenge here too!
>> > > >> > > > >> >
>> > > >> > > > >> > Let's look at improving Load-balancing
offering from
>> > > >> cloudstack, I
>> > > >> > > guest
>> > > >> > > > >> we
>> > > >> > > > >> > should do a feature spec draft soon..,
 from my
>> perspective,
>> > > >> doing
>> > > >> > > SSL
>> > > >> > > > >> > offload on the VR could be problematic
if the VR spec if
>> too
>> > > >> small,
>> > > >> > > and
>> > > >> > > > >> the
>> > > >> > > > >> > default spec of the VR being 1vcpu@256MB,
considering
>> it can
>> > > >> be the
>> > > >> > > > >> router
>> > > >> > > > >> > of a VPC, doing VPN termination,
adding HTTPS  is a bit
>> > > ish...
>> > > >> What
>> > > >> > > would
>> > > >> > > > >> > be your thought about this ?
>> > > >> > > > >> >
>> > > >> > > > >> > I'd be curious to have a LB offering
in ACS where it
>> would
>> > > >> deploy a
>> > > >> > > > >> > redundant traefik[1] beside the VR
for doing http and
>> https
>> > > >> > > > >> Load-balancing.
>> > > >> > > > >> > I think it would also be useful if
the API of that
>> traefik
>> > > >> instance
>> > > >> > > would
>> > > >> > > > >> > be available from within the VPC
or LBnetwork so is API
>> > > would be
>> > > >> > > > >> accessible
>> > > >> > > > >> > to other apps orchestration tools
such as  kubernetes or
>> > > >> rancher.
>> > > >> > > > >> >
>> > > >> > > > >> > traefik or not, here is what I think
is needed by
>> cloudstack
>> > > in
>> > > >> the
>> > > >> > > LB
>> > > >> > > > >> > improvement:
>> > > >> > > > >> >
>> > > >> > > > >> > - support http, https (X-Forwarded-For)
>> > > >> > > > >>
>> > > >> > > > >> HAProxy also supports the PROXY protocol
towards the
>> backends.
>> > > >> Apache
>> > > >> > > > >> 2.4.22 supports this natively and Varnish
for example can
>> also
>> > > >> talk
>> > > >> > > PROXY.
>> > > >> > > > >>
>> > > >> > > > >> It adds a littlebit of metadata to the
connection so that
>> the
>> > > >> backend
>> > > >> > > > >> knows the original IP the connection came
from for example:
>> > > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-
>> protocol.txt
>> > > >> > > > >>
>> > > >> > > > >> Wido
>> > > >> > > > >>
>> > > >> > > > >> > - basic persistence tuning (API already
exist)
>> > > >> > > > >> > - better backend monitoring, currently
only a tcp connect
>> > > >> validate
>> > > >> > > if the
>> > > >> > > > >> > webserver is up.
>> > > >> > > > >> > - ssl offload
>> > > >> > > > >> > - metric collection, more stats,
maybe just export the
>> tool
>> > > >> status
>> > > >> > > page
>> > > >> > > > >> to
>> > > >> > > > >> > the private network.
>> > > >> > > > >> > - Container world support, right
now if you have Rancher
>> or
>> > > >> > > kubernetes
>> > > >> > > > >> > cluster, you need to deploy your
own LB solution behing
>> > > >> mostlikely a
>> > > >> > > > >> static
>> > > >> > > > >> > nat., If cloudstack would deploy
a traefik instance, Kub
>> or
>> > > >> Rancher
>> > > >> > > could
>> > > >> > > > >> > reuse this instance and managed it
to properly do LB
>> between
>> > > >> > > containers.
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > What would be your prefered LB tool:
>> > > >> > > > >> > haproxy, traefik or nginx?
>> > > >> > > > >> >
>> > > >> > > > >> > CloudStack already have to code to
handle SSL certs per
>> > > >> projects and
>> > > >> > > > >> > accounts if not mistaking because
that code was added to
>> > > support
>> > > >> > > > >> NetScaler
>> > > >> > > > >> > as Load-balancer in the past. so
one less thing to think
>> > > about
>> > > >> :-)
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > [1] https://traefik.io/
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > PL,
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> >
>> > > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux!
<nux@li.nux.ro>
>> wrote:
>> > > >> > > > >> >
>> > > >> > > > >> > > Thanks Andrija,
>> > > >> > > > >> > >
>> > > >> > > > >> > > LB outside of the VR sounds
like a good idea. An
>> appliance
>> > > >> based
>> > > >> > > on,
>> > > >> > > > >> say
>> > > >> > > > >> > > cloud-init + ansible and so
on could do the trick; alas
>> > > it'd
>> > > >> need
>> > > >> > > to be
>> > > >> > > > >> > > outside ACS.
>> > > >> > > > >> > > I guess as users we could maybe
come up with a spec
>> for an
>> > > >> > > > >> improvement, at
>> > > >> > > > >> > > least we'd have something the
devs could look at
>> whenever
>> > > it
>> > > >> is
>> > > >> > > > >> possible.
>> > > >> > > > >> > >
>> > > >> > > > >> > > Regards,
>> > > >> > > > >> > > Lucian
>> > > >> > > > >> > >
>> > > >> > > > >> > > --
>> > > >> > > > >> > > Sent from the Delta quadrant
using Borg technology!
>> > > >> > > > >> > >
>> > > >> > > > >> > > Nux!
>> > > >> > > > >> > > www.nux.ro
>> > > >> > > > >> > >
>> > > >> > > > >> > > ----- Original Message -----
>> > > >> > > > >> > > > From: "Andrija Panic" <andrija.panic@gmail.com>
>> > > >> > > > >> > > > To: "dev" <dev@cloudstack.apache.org>
>> > > >> > > > >> > > > Cc: "users" <users@cloudstack.apache.org>
>> > > >> > > > >> > > > Sent: Thursday, 2 November,
2017 23:21:37
>> > > >> > > > >> > > > Subject: Re: HTTPS LB and
x-forwarded-for
>> > > >> > > > >> > >
>> > > >> > > > >> > > > We used to make some special
stuff for one of the
>> > > clients,
>> > > >> > > where all
>> > > >> > > > >> LB
>> > > >> > > > >> > > > configuration work is done
from outside of the ACS,
>> i.e.
>> > > >> python
>> > > >> > > > >> script to
>> > > >> > > > >> > > > feed/configure VR - install
latest haproxy 1.5.x for
>> > > >> transparent
>> > > >> > > > >> proxy,
>> > > >> > > > >> > > > since client insisted on
SSL termination done on
>> backend
>> > > >> web SSL
>> > > >> > > > >> > > servers....
>> > > >> > > > >> > > > Not good idea, that is
all I can say (custom
>> > > configuration
>> > > >> > > thing) -
>> > > >> > > > >> but
>> > > >> > > > >> > > the
>> > > >> > > > >> > > > LB setup is actually good
- transparent mode haproxy,
>> > > works
>> > > >> on
>> > > >> > > TCP
>> > > >> > > > >> level,
>> > > >> > > > >> > > > so you can see "real client
IP" on the backend
>> servers
>> > > >> (which
>> > > >> > > must
>> > > >> > > > >> use VR
>> > > >> > > > >> > > > as the default gtw, as
per default, so the whole
>> setup
>> > > works
>> > > >> > > > >> properly).
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > I'm still looking forward
to see some special
>> support of
>> > > LB
>> > > >> > > inside
>> > > >> > > > >> VR via
>> > > >> > > > >> > > > ACS - proper LB setup inside
VR via GUI/API -  i.e.
>> to
>> > > >> enable LB
>> > > >> > > > >> > > > provisioning SCRIPT (bash,
or whatever),  where all
>> > > needed
>> > > >> > > > >> > > > install+configure can be
done from client side  -
>> > > otherwise
>> > > >> > > covering
>> > > >> > > > >> all
>> > > >> > > > >> > > > user cases, with proper
HTTP checks and similar....is
>> > > >> > > impossible to
>> > > >> > > > >> do
>> > > >> > > > >> > > > IMHO.
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > Some other clients, actually
have internal FW
>> appliance
>> > > >> (i.e.
>> > > >> > > > >> multihomed
>> > > >> > > > >> > > > VM, acting as gtw for all
VMs in all networks), and
>> > > haproxy
>> > > >> > > instaled
>> > > >> > > > >> on
>> > > >> > > > >> > > > this device (with NAT configured
from VR to this
>> internal
>> > > >> > > FW/VM, so
>> > > >> > > > >> > > remote
>> > > >> > > > >> > > > IP can be seen properly)
- this setup is fully under
>> > > >> customer
>> > > >> > > > >> control,
>> > > >> > > > >> > > and
>> > > >> > > > >> > > > can provide any kind of
special haproxy config...
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > On 31 October 2017 at 19:54,
Nux! <nux@li.nux.ro>
>> wrote:
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >> Hello,
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Of the people running
an LB (VR) with https
>> backends,
>> > > how
>> > > >> do
>> > > >> > > you
>> > > >> > > > >> deal
>> > > >> > > > >> > > with
>> > > >> > > > >> > > >> the lack of x-forwarded-for
since for port 443
>> there's
>> > > just
>> > > >> > > simple
>> > > >> > > > >> TCP
>> > > >> > > > >> > > >> balancing?
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Has anyone thought
of terminating SSL in the VR
>> instead?
>> > > >> Ideas?
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Cheers
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> --
>> > > >> > > > >> > > >> Sent from the Delta
quadrant using Borg technology!
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >> Nux!
>> > > >> > > > >> > > >> www.nux.ro
>> > > >> > > > >> > > >>
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > --
>> > > >> > > > >> > > >
>> > > >> > > > >> > > > Andrija Panić
>> > > >> > > > >> > >
>> > > >> > > > >>
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > --
>> > > >> > > > >
>> > > >> > > > > Andrija Panić
>> > > >> > >
>> > >

Mime
View raw message