Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Date: Mon, 13 Nov 2017 15:18:42 +0100 (CET)
From: Wido den Hollander <wido@widodh.nl>
To: dev@cloudstack.apache.org, users@cloudstack.apache.org,
	Pierre-Luc Dion <pdion@cloudops.com>
Message-ID: <155403812.4713.1510582722328@ox.pcextreme.nl>
In-Reply-To: <CAD97ioc-OkEd+-J=vwKfeiCwHftLwAZ5SdO=g7QhtD=W_RQbew@mail.gmail.com>
References: <1750999994.6369.1509476065042.JavaMail.zimbra@li.nux.ro>
 <1778806379.4501.1510211582337@ox.pcextreme.nl>
 <CAMvtBPNrWQGphKyQixJQCPbRLDODzTx_Ct4YQ4zd8ia1gnHVGQ@mail.gmail.com>
 <1657929870.1138.1510253995466.JavaMail.zimbra@li.nux.ro>
 <940185856.4522.1510296264699@ox.pcextreme.nl>
 <CAD97iodfuN85Lkc5eLhBuusXWsZmqcy1w4Fzba3EY6mjsdb4Qw@mail.gmail.com>
 <580263127.4633.1510324476134@ox.pcextreme.nl>
 <CAD97iodBWYfb+8Bq3=FKmnK76xqkOTs2fn_cAtMR1CQot3zOEA@mail.gmail.com>
 <2102575241.1695.1510328187286.JavaMail.zimbra@li.nux.ro>
 <CAD97ioc-OkEd+-J=vwKfeiCwHftLwAZ5SdO=g7QhtD=W_RQbew@mail.gmail.com>
Subject: Re: HTTPS LB and x-forwarded-for
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Importance: Medium
archived-at: Mon, 13 Nov 2017 14:19:13 -0000


> Op 13 november 2017 om 15:14 schreef Pierre-Luc Dion <pdion@cloudops.com>=
:
>=20
>=20
> Hi,
>=20
> This is looking quite promising, I have a colleague that tested that last
> Friday, look like the proxy proto v1 work out of the box with Nginx, but
> would need an extra package for Apache 2.4 ?

It depends. You need HTTPd 2.4.28, see: https://httpd.apache.org/docs/trunk=
/mod/mod_remoteip.html#remoteipproxyprotocol

It's there upstream, but not in all packages.=20

It can from this module:

- https://github.com/roadrunner2/mod-proxy-protocol
- https://roadrunner2.github.io/mod-proxy-protocol/mod_proxy_protocol.html

They donated the code to go upstream and went into mod_remoteip but landed =
in 2.4.28

It will probably make it into Ubuntu 18.04 and CentOS 7.4.

Wido

> Otherwise, it seems to be a good way to do  https LB without complicated
> configuration and huge changes in CloudStack.
>=20
>=20
>=20
> On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nux@li.nux.ro> wrote:
>=20
> > Pierre-Luc,
> >
> > Haproxy docs say it should work for any kind of traffic as long as both
> > ends are PROXY-aware and it look like a majority of software is.
> > So, in short, yes.
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Pierre-Luc Dion" <pdion@cloudops.com>
> > > To: "Wido den Hollander" <wido@widodh.nl>
> > > Cc: "dev" <dev@cloudstack.apache.org>, "Khosrow Moossavi" <
> > kmoossavi@cloudops.com>, "Will Stevens"
> > > <wstevens@cloudops.com>, "Nux!" <nux@li.nux.ro>, "users" <
> > users@cloudstack.apache.org>
> > > Sent: Friday, 10 November, 2017 15:32:38
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > Hi Wido, do you know if this would work for https traffic too?
> > >
> > > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wido@widodh.nl> a =C3=
=A9crit :
> > >
> > >>
> > >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> > pdion@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > I kind of like the proxy backend type, ill check on our end if tha=
t
> > would
> > >> > work but definitely a simple and efficient approach!
> > >> >
> > >>
> > >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> > >>
> > >> Apache HTTPd supports PROXY since 2.4.28:
> > https://httpd.apache.org/docs/
> > >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> > >>
> > >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> > >>
> > >> Wido
> > >>
> > >> >
> > >> >
> > >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wido@widodh.nl> a
> > =C3=A9crit :
> > >> >
> > >> > >
> > >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nux@li.nux.ro>:
> > >> > > >
> > >> > > >
> > >> > > > Wido,
> > >> > > >
> > >> > > > Excellent suggestion with the "transparent proxy", I was not
> > aware of
> > >> > > that.
> > >> > > > I think that would be a great idea and wouldn't require too ma=
ny
> > >> > > modifications, especially as Haproxy comes already with the VR.
> > >> > > >
> > >> > >
> > >> > > It's indeed just a matter of a HAProxy config setting. We could
> > make it
> > >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY =
for
> > >> example.
> > >> > >
> > >> > > That way your problem would be solved.
> > >> > >
> > >> > > Wido
> > >> > >
> > >> > > > To Paul:
> > >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> > you do
> > >> > > not know the remote host ip. You're flying blind unless you use
> > google
> > >> > > analytics (and these things have gotten more and more aggressive=
ly
> > >> filtered
> > >> > > by adblocks).
> > >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> > wouldn't
> > >> > > break existing functionality and would also keep SSL processing =
off
> > >> the VR.
> > >> > > >
> > >> > > > --
> > >> > > > Sent from the Delta quadrant using Borg technology!
> > >> > > >
> > >> > > > Nux!
> > >> > > > www.nux.ro
> > >> > > >
> > >> > > > ----- Original Message -----
> > >> > > > > From: "Andrija Panic" <andrija.panic@gmail.com>
> > >> > > > > To: "users" <users@cloudstack.apache.org>
> > >> > > > > Cc: "Khosrow Moossavi" <kmoossavi@cloudops.com>, "Will
> > Stevens" <
> > >> > > wstevens@cloudops.com>, "dev"
> > >> > > > > <dev@cloudstack.apache.org>, "Pierre-Luc Dion" <
> > pdion@cloudops.com
> > >> >
> > >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > > >
> > >> > > > > Wido,
> > >> > > > >
> > >> > > > > backend servers are not Linux only, for example we have a to=
n of
> > >> > > Windows
> > >> > > > > customers, some WEB solutions / IIS etc...
> > >> > > > >
> > >> > > > > @all - If we try to please/solve everyone's proxying
> > >> > > solution/requirement -
> > >> > > > > this is impossible IMHO - I'm thinking more about some "do i=
t as
> > >> you
> > >> > > like"
> > >> > > > > solution, to let customer write his own haproxy config and
> > upoad it
> > >> > > (for
> > >> > > > > example, or something better?).
> > >> > > > >
> > >> > > > > We can support newer version of haproxy (1.5+) which also
> > implement
> > >> > > > > "transarent proxy" (integrate with kernel so to speak)  to a=
llow
> > >> > > TCP-level
> > >> > > > > connections to backend (TCP mode, not HTTP mode) but to stil=
l
> > >> > > "preserve"
> > >> > > > > remote IP by faking it (fake soruce IP =3D transarent proxy)=
.
> > >> > > > >
> > >> > > > > For the rest of configuration options,  I would leave it to =
the
> > >> > > customer
> > >> > > > > how he/she wants to configure rest of haproxy configuration,
> > >> inlcuding
> > >> > > > > custom checks, etc. Haproxy configuration is never-ending st=
ory,
> > >> and we
> > >> > > > > probably should allow custom sripts/configuration instead of
> > >> trying to
> > >> > > > > provide GUI/API way to configure everything (which is
> > >> impossible...)
> > >> > > > >
> > >> > > > > Just my 2 cents...
> > >> > > > >
> > >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh=
.nl
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > >>
> > >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > >> > > pdion@cloudops.com
> > >> > > > >> >:
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > Same challenge here too!
> > >> > > > >> >
> > >> > > > >> > Let's look at improving Load-balancing offering from
> > >> cloudstack, I
> > >> > > guest
> > >> > > > >> we
> > >> > > > >> > should do a feature spec draft soon..,  from my perspecti=
ve,
> > >> doing
> > >> > > SSL
> > >> > > > >> > offload on the VR could be problematic if the VR spec if =
too
> > >> small,
> > >> > > and
> > >> > > > >> the
> > >> > > > >> > default spec of the VR being 1vcpu@256MB, considering it =
can
> > >> be the
> > >> > > > >> router
> > >> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit
> > ish...
> > >> What
> > >> > > would
> > >> > > > >> > be your thought about this ?
> > >> > > > >> >
> > >> > > > >> > I'd be curious to have a LB offering in ACS where it woul=
d
> > >> deploy a
> > >> > > > >> > redundant traefik[1] beside the VR for doing http and htt=
ps
> > >> > > > >> Load-balancing.
> > >> > > > >> > I think it would also be useful if the API of that traefi=
k
> > >> instance
> > >> > > would
> > >> > > > >> > be available from within the VPC or LBnetwork so is API
> > would be
> > >> > > > >> accessible
> > >> > > > >> > to other apps orchestration tools such as  kubernetes or
> > >> rancher.
> > >> > > > >> >
> > >> > > > >> > traefik or not, here is what I think is needed by cloudst=
ack
> > in
> > >> the
> > >> > > LB
> > >> > > > >> > improvement:
> > >> > > > >> >
> > >> > > > >> > - support http, https (X-Forwarded-For)
> > >> > > > >>
> > >> > > > >> HAProxy also supports the PROXY protocol towards the backen=
ds.
> > >> Apache
> > >> > > > >> 2.4.22 supports this natively and Varnish for example can a=
lso
> > >> talk
> > >> > > PROXY.
> > >> > > > >>
> > >> > > > >> It adds a littlebit of metadata to the connection so that t=
he
> > >> backend
> > >> > > > >> knows the original IP the connection came from for example:
> > >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > >> > > > >>
> > >> > > > >> Wido
> > >> > > > >>
> > >> > > > >> > - basic persistence tuning (API already exist)
> > >> > > > >> > - better backend monitoring, currently only a tcp connect
> > >> validate
> > >> > > if the
> > >> > > > >> > webserver is up.
> > >> > > > >> > - ssl offload
> > >> > > > >> > - metric collection, more stats, maybe just export the to=
ol
> > >> status
> > >> > > page
> > >> > > > >> to
> > >> > > > >> > the private network.
> > >> > > > >> > - Container world support, right now if you have Rancher =
or
> > >> > > kubernetes
> > >> > > > >> > cluster, you need to deploy your own LB solution behing
> > >> mostlikely a
> > >> > > > >> static
> > >> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub =
or
> > >> Rancher
> > >> > > could
> > >> > > > >> > reuse this instance and managed it to properly do LB betw=
een
> > >> > > containers.
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > What would be your prefered LB tool:
> > >> > > > >> > haproxy, traefik or nginx?
> > >> > > > >> >
> > >> > > > >> > CloudStack already have to code to handle SSL certs per
> > >> projects and
> > >> > > > >> > accounts if not mistaking because that code was added to
> > support
> > >> > > > >> NetScaler
> > >> > > > >> > as Load-balancer in the past. so one less thing to think
> > about
> > >> :-)
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > [1] https://traefik.io/
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > PL,
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nux@li.nux.ro> wrot=
e:
> > >> > > > >> >
> > >> > > > >> > > Thanks Andrija,
> > >> > > > >> > >
> > >> > > > >> > > LB outside of the VR sounds like a good idea. An applia=
nce
> > >> based
> > >> > > on,
> > >> > > > >> say
> > >> > > > >> > > cloud-init + ansible and so on could do the trick; alas
> > it'd
> > >> need
> > >> > > to be
> > >> > > > >> > > outside ACS.
> > >> > > > >> > > I guess as users we could maybe come up with a spec for=
 an
> > >> > > > >> improvement, at
> > >> > > > >> > > least we'd have something the devs could look at whenev=
er
> > it
> > >> is
> > >> > > > >> possible.
> > >> > > > >> > >
> > >> > > > >> > > Regards,
> > >> > > > >> > > Lucian
> > >> > > > >> > >
> > >> > > > >> > > --
> > >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > >> > > > >> > >
> > >> > > > >> > > Nux!
> > >> > > > >> > > www.nux.ro
> > >> > > > >> > >
> > >> > > > >> > > ----- Original Message -----
> > >> > > > >> > > > From: "Andrija Panic" <andrija.panic@gmail.com>
> > >> > > > >> > > > To: "dev" <dev@cloudstack.apache.org>
> > >> > > > >> > > > Cc: "users" <users@cloudstack.apache.org>
> > >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> > >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >> > > > >> > >
> > >> > > > >> > > > We used to make some special stuff for one of the
> > clients,
> > >> > > where all
> > >> > > > >> LB
> > >> > > > >> > > > configuration work is done from outside of the ACS, i=
.e.
> > >> python
> > >> > > > >> script to
> > >> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
> > >> transparent
> > >> > > > >> proxy,
> > >> > > > >> > > > since client insisted on SSL termination done on back=
end
> > >> web SSL
> > >> > > > >> > > servers....
> > >> > > > >> > > > Not good idea, that is all I can say (custom
> > configuration
> > >> > > thing) -
> > >> > > > >> but
> > >> > > > >> > > the
> > >> > > > >> > > > LB setup is actually good - transparent mode haproxy,
> > works
> > >> on
> > >> > > TCP
> > >> > > > >> level,
> > >> > > > >> > > > so you can see "real client IP" on the backend server=
s
> > >> (which
> > >> > > must
> > >> > > > >> use VR
> > >> > > > >> > > > as the default gtw, as per default, so the whole setu=
p
> > works
> > >> > > > >> properly).
> > >> > > > >> > > >
> > >> > > > >> > > > I'm still looking forward to see some special support=
 of
> > LB
> > >> > > inside
> > >> > > > >> VR via
> > >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. t=
o
> > >> enable LB
> > >> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all
> > needed
> > >> > > > >> > > > install+configure can be done from client side  -
> > otherwise
> > >> > > covering
> > >> > > > >> all
> > >> > > > >> > > > user cases, with proper HTTP checks and similar....is
> > >> > > impossible to
> > >> > > > >> do
> > >> > > > >> > > > IMHO.
> > >> > > > >> > > >
> > >> > > > >> > > > Some other clients, actually have internal FW applian=
ce
> > >> (i.e.
> > >> > > > >> multihomed
> > >> > > > >> > > > VM, acting as gtw for all VMs in all networks), and
> > haproxy
> > >> > > instaled
> > >> > > > >> on
> > >> > > > >> > > > this device (with NAT configured from VR to this inte=
rnal
> > >> > > FW/VM, so
> > >> > > > >> > > remote
> > >> > > > >> > > > IP can be seen properly) - this setup is fully under
> > >> customer
> > >> > > > >> control,
> > >> > > > >> > > and
> > >> > > > >> > > > can provide any kind of special haproxy config...
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nux@li.nux.ro> wro=
te:
> > >> > > > >> > > >
> > >> > > > >> > > >> Hello,
> > >> > > > >> > > >>
> > >> > > > >> > > >> Of the people running an LB (VR) with https backends=
,
> > how
> > >> do
> > >> > > you
> > >> > > > >> deal
> > >> > > > >> > > with
> > >> > > > >> > > >> the lack of x-forwarded-for since for port 443 there=
's
> > just
> > >> > > simple
> > >> > > > >> TCP
> > >> > > > >> > > >> balancing?
> > >> > > > >> > > >>
> > >> > > > >> > > >> Has anyone thought of terminating SSL in the VR inst=
ead?
> > >> Ideas?
> > >> > > > >> > > >>
> > >> > > > >> > > >> Cheers
> > >> > > > >> > > >>
> > >> > > > >> > > >> --
> > >> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
> > >> > > > >> > > >>
> > >> > > > >> > > >> Nux!
> > >> > > > >> > > >> www.nux.ro
> > >> > > > >> > > >>
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > >
> > >> > > > >> > > > --
> > >> > > > >> > > >
> > >> > > > >> > > > Andrija Pani=C4=87
> > >> > > > >> > >
> > >> > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > >
> > >> > > > > Andrija Pani=C4=87
> > >> > >
> >