Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Date: Fri, 10 Nov 2017 15:36:27 +0000 (GMT)
From: Nux! <nux@li.nux.ro>
To: dev <dev@cloudstack.apache.org>
Cc: Wido den Hollander <wido@widodh.nl>,
	Khosrow Moossavi <kmoossavi@cloudops.com>,
	Will Stevens <wstevens@cloudops.com>,
	users <users@cloudstack.apache.org>
Message-ID: <2102575241.1695.1510328187286.JavaMail.zimbra@li.nux.ro>
In-Reply-To: <CAD97iodBWYfb+8Bq3=FKmnK76xqkOTs2fn_cAtMR1CQot3zOEA@mail.gmail.com>
References: <1750999994.6369.1509476065042.JavaMail.zimbra@li.nux.ro> <1778806379.4501.1510211582337@ox.pcextreme.nl> <CAMvtBPNrWQGphKyQixJQCPbRLDODzTx_Ct4YQ4zd8ia1gnHVGQ@mail.gmail.com> <1657929870.1138.1510253995466.JavaMail.zimbra@li.nux.ro> <940185856.4522.1510296264699@ox.pcextreme.nl> <CAD97iodfuN85Lkc5eLhBuusXWsZmqcy1w4Fzba3EY6mjsdb4Qw@mail.gmail.com> <580263127.4633.1510324476134@ox.pcextreme.nl> <CAD97iodBWYfb+8Bq3=FKmnK76xqkOTs2fn_cAtMR1CQot3zOEA@mail.gmail.com>
Subject: Re: HTTPS LB and x-forwarded-for
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Thread-Topic: HTTPS LB and x-forwarded-for
Thread-Index: PjD/Mj5f6g+QptlBUIXZ2mqz3qXeZw==
archived-at: Fri, 10 Nov 2017 15:36:43 -0000

Pierre-Luc,

Haproxy docs say it should work for any kind of traffic as long as both end=
s are PROXY-aware and it look like a majority of software is.
So, in short, yes.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Pierre-Luc Dion" <pdion@cloudops.com>
> To: "Wido den Hollander" <wido@widodh.nl>
> Cc: "dev" <dev@cloudstack.apache.org>, "Khosrow Moossavi" <kmoossavi@clou=
dops.com>, "Will Stevens"
> <wstevens@cloudops.com>, "Nux!" <nux@li.nux.ro>, "users" <users@cloudstac=
k.apache.org>
> Sent: Friday, 10 November, 2017 15:32:38
> Subject: Re: HTTPS LB and x-forwarded-for

> Hi Wido, do you know if this would work for https traffic too?
>=20
> Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wido@widodh.nl> a =C3=A9cr=
it :
>=20
>>
>> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <pdion@cloudops.c=
om
>> >:
>> >
>> >
>> > I kind of like the proxy backend type, ill check on our end if that wo=
uld
>> > work but definitely a simple and efficient approach!
>> >
>>
>> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>>
>> Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/
>> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>>
>> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>>
>> Wido
>>
>> >
>> >
>> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wido@widodh.nl> a =C3=
=A9crit :
>> >
>> > >
>> > > > Op 9 november 2017 om 19:59 schreef Nux! <nux@li.nux.ro>:
>> > > >
>> > > >
>> > > > Wido,
>> > > >
>> > > > Excellent suggestion with the "transparent proxy", I was not aware=
 of
>> > > that.
>> > > > I think that would be a great idea and wouldn't require too many
>> > > modifications, especially as Haproxy comes already with the VR.
>> > > >
>> > >
>> > > It's indeed just a matter of a HAProxy config setting. We could make=
 it
>> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
>> example.
>> > >
>> > > That way your problem would be solved.
>> > >
>> > > Wido
>> > >
>> > > > To Paul:
>> > > > - imho the LB solution ACS ships now is a bit handicaped since you=
 do
>> > > not know the remote host ip. You're flying blind unless you use goog=
le
>> > > analytics (and these things have gotten more and more aggressively
>> filtered
>> > > by adblocks).
>> > > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn=
't
>> > > break existing functionality and would also keep SSL processing off
>> the VR.
>> > > >
>> > > > --
>> > > > Sent from the Delta quadrant using Borg technology!
>> > > >
>> > > > Nux!
>> > > > www.nux.ro
>> > > >
>> > > > ----- Original Message -----
>> > > > > From: "Andrija Panic" <andrija.panic@gmail.com>
>> > > > > To: "users" <users@cloudstack.apache.org>
>> > > > > Cc: "Khosrow Moossavi" <kmoossavi@cloudops.com>, "Will Stevens" =
<
>> > > wstevens@cloudops.com>, "dev"
>> > > > > <dev@cloudstack.apache.org>, "Pierre-Luc Dion" <pdion@cloudops.c=
om
>> >
>> > > > > Sent: Thursday, 9 November, 2017 13:10:58
>> > > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >
>> > > > > Wido,
>> > > > >
>> > > > > backend servers are not Linux only, for example we have a ton of
>> > > Windows
>> > > > > customers, some WEB solutions / IIS etc...
>> > > > >
>> > > > > @all - If we try to please/solve everyone's proxying
>> > > solution/requirement -
>> > > > > this is impossible IMHO - I'm thinking more about some "do it as
>> you
>> > > like"
>> > > > > solution, to let customer write his own haproxy config and upoad=
 it
>> > > (for
>> > > > > example, or something better?).
>> > > > >
>> > > > > We can support newer version of haproxy (1.5+) which also implem=
ent
>> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
>> > > TCP-level
>> > > > > connections to backend (TCP mode, not HTTP mode) but to still
>> > > "preserve"
>> > > > > remote IP by faking it (fake soruce IP =3D transarent proxy).
>> > > > >
>> > > > > For the rest of configuration options,  I would leave it to the
>> > > customer
>> > > > > how he/she wants to configure rest of haproxy configuration,
>> inlcuding
>> > > > > custom checks, etc. Haproxy configuration is never-ending story,
>> and we
>> > > > > probably should allow custom sripts/configuration instead of
>> trying to
>> > > > > provide GUI/API way to configure everything (which is
>> impossible...)
>> > > > >
>> > > > > Just my 2 cents...
>> > > > >
>> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh.nl>
>> > > wrote:
>> > > > >
>> > > > >>
>> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
>> > > pdion@cloudops.com
>> > > > >> >:
>> > > > >> >
>> > > > >> >
>> > > > >> > Same challenge here too!
>> > > > >> >
>> > > > >> > Let's look at improving Load-balancing offering from
>> cloudstack, I
>> > > guest
>> > > > >> we
>> > > > >> > should do a feature spec draft soon..,  from my perspective,
>> doing
>> > > SSL
>> > > > >> > offload on the VR could be problematic if the VR spec if too
>> small,
>> > > and
>> > > > >> the
>> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
>> be the
>> > > > >> router
>> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish..=
.
>> What
>> > > would
>> > > > >> > be your thought about this ?
>> > > > >> >
>> > > > >> > I'd be curious to have a LB offering in ACS where it would
>> deploy a
>> > > > >> > redundant traefik[1] beside the VR for doing http and https
>> > > > >> Load-balancing.
>> > > > >> > I think it would also be useful if the API of that traefik
>> instance
>> > > would
>> > > > >> > be available from within the VPC or LBnetwork so is API would=
 be
>> > > > >> accessible
>> > > > >> > to other apps orchestration tools such as  kubernetes or
>> rancher.
>> > > > >> >
>> > > > >> > traefik or not, here is what I think is needed by cloudstack =
in
>> the
>> > > LB
>> > > > >> > improvement:
>> > > > >> >
>> > > > >> > - support http, https (X-Forwarded-For)
>> > > > >>
>> > > > >> HAProxy also supports the PROXY protocol towards the backends.
>> Apache
>> > > > >> 2.4.22 supports this natively and Varnish for example can also
>> talk
>> > > PROXY.
>> > > > >>
>> > > > >> It adds a littlebit of metadata to the connection so that the
>> backend
>> > > > >> knows the original IP the connection came from for example:
>> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>> > > > >>
>> > > > >> Wido
>> > > > >>
>> > > > >> > - basic persistence tuning (API already exist)
>> > > > >> > - better backend monitoring, currently only a tcp connect
>> validate
>> > > if the
>> > > > >> > webserver is up.
>> > > > >> > - ssl offload
>> > > > >> > - metric collection, more stats, maybe just export the tool
>> status
>> > > page
>> > > > >> to
>> > > > >> > the private network.
>> > > > >> > - Container world support, right now if you have Rancher or
>> > > kubernetes
>> > > > >> > cluster, you need to deploy your own LB solution behing
>> mostlikely a
>> > > > >> static
>> > > > >> > nat., If cloudstack would deploy a traefik instance, Kub or
>> Rancher
>> > > could
>> > > > >> > reuse this instance and managed it to properly do LB between
>> > > containers.
>> > > > >> >
>> > > > >> >
>> > > > >> > What would be your prefered LB tool:
>> > > > >> > haproxy, traefik or nginx?
>> > > > >> >
>> > > > >> > CloudStack already have to code to handle SSL certs per
>> projects and
>> > > > >> > accounts if not mistaking because that code was added to supp=
ort
>> > > > >> NetScaler
>> > > > >> > as Load-balancer in the past. so one less thing to think abou=
t
>> :-)
>> > > > >> >
>> > > > >> >
>> > > > >> > [1] https://traefik.io/
>> > > > >> >
>> > > > >> >
>> > > > >> > PL,
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nux@li.nux.ro> wrote:
>> > > > >> >
>> > > > >> > > Thanks Andrija,
>> > > > >> > >
>> > > > >> > > LB outside of the VR sounds like a good idea. An appliance
>> based
>> > > on,
>> > > > >> say
>> > > > >> > > cloud-init + ansible and so on could do the trick; alas it'=
d
>> need
>> > > to be
>> > > > >> > > outside ACS.
>> > > > >> > > I guess as users we could maybe come up with a spec for an
>> > > > >> improvement, at
>> > > > >> > > least we'd have something the devs could look at whenever i=
t
>> is
>> > > > >> possible.
>> > > > >> > >
>> > > > >> > > Regards,
>> > > > >> > > Lucian
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Sent from the Delta quadrant using Borg technology!
>> > > > >> > >
>> > > > >> > > Nux!
>> > > > >> > > www.nux.ro
>> > > > >> > >
>> > > > >> > > ----- Original Message -----
>> > > > >> > > > From: "Andrija Panic" <andrija.panic@gmail.com>
>> > > > >> > > > To: "dev" <dev@cloudstack.apache.org>
>> > > > >> > > > Cc: "users" <users@cloudstack.apache.org>
>> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > > >> > >
>> > > > >> > > > We used to make some special stuff for one of the clients=
,
>> > > where all
>> > > > >> LB
>> > > > >> > > > configuration work is done from outside of the ACS, i.e.
>> python
>> > > > >> script to
>> > > > >> > > > feed/configure VR - install latest haproxy 1.5.x for
>> transparent
>> > > > >> proxy,
>> > > > >> > > > since client insisted on SSL termination done on backend
>> web SSL
>> > > > >> > > servers....
>> > > > >> > > > Not good idea, that is all I can say (custom configuratio=
n
>> > > thing) -
>> > > > >> but
>> > > > >> > > the
>> > > > >> > > > LB setup is actually good - transparent mode haproxy, wor=
ks
>> on
>> > > TCP
>> > > > >> level,
>> > > > >> > > > so you can see "real client IP" on the backend servers
>> (which
>> > > must
>> > > > >> use VR
>> > > > >> > > > as the default gtw, as per default, so the whole setup wo=
rks
>> > > > >> properly).
>> > > > >> > > >
>> > > > >> > > > I'm still looking forward to see some special support of =
LB
>> > > inside
>> > > > >> VR via
>> > > > >> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to
>> enable LB
>> > > > >> > > > provisioning SCRIPT (bash, or whatever),  where all neede=
d
>> > > > >> > > > install+configure can be done from client side  - otherwi=
se
>> > > covering
>> > > > >> all
>> > > > >> > > > user cases, with proper HTTP checks and similar....is
>> > > impossible to
>> > > > >> do
>> > > > >> > > > IMHO.
>> > > > >> > > >
>> > > > >> > > > Some other clients, actually have internal FW appliance
>> (i.e.
>> > > > >> multihomed
>> > > > >> > > > VM, acting as gtw for all VMs in all networks), and hapro=
xy
>> > > instaled
>> > > > >> on
>> > > > >> > > > this device (with NAT configured from VR to this internal
>> > > FW/VM, so
>> > > > >> > > remote
>> > > > >> > > > IP can be seen properly) - this setup is fully under
>> customer
>> > > > >> control,
>> > > > >> > > and
>> > > > >> > > > can provide any kind of special haproxy config...
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nux@li.nux.ro> wrote:
>> > > > >> > > >
>> > > > >> > > >> Hello,
>> > > > >> > > >>
>> > > > >> > > >> Of the people running an LB (VR) with https backends, ho=
w
>> do
>> > > you
>> > > > >> deal
>> > > > >> > > with
>> > > > >> > > >> the lack of x-forwarded-for since for port 443 there's j=
ust
>> > > simple
>> > > > >> TCP
>> > > > >> > > >> balancing?
>> > > > >> > > >>
>> > > > >> > > >> Has anyone thought of terminating SSL in the VR instead?
>> Ideas?
>> > > > >> > > >>
>> > > > >> > > >> Cheers
>> > > > >> > > >>
>> > > > >> > > >> --
>> > > > >> > > >> Sent from the Delta quadrant using Borg technology!
>> > > > >> > > >>
>> > > > >> > > >> Nux!
>> > > > >> > > >> www.nux.ro
>> > > > >> > > >>
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > > > --
>> > > > >> > > >
>> > > > >> > > > Andrija Pani=C4=87
>> > > > >> > >
>> > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > Andrija Pani=C4=87
>> > >