Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Date: Thu, 9 Nov 2017 18:59:55 +0000 (GMT)
From: Nux! <nux@li.nux.ro>
To: users <users@cloudstack.apache.org>
Cc: Khosrow Moossavi <kmoossavi@cloudops.com>,
	Will Stevens <wstevens@cloudops.com>,
	dev <dev@cloudstack.apache.org>,
	Pierre-Luc Dion <pdion@cloudops.com>
Message-ID: <1657929870.1138.1510253995466.JavaMail.zimbra@li.nux.ro>
In-Reply-To: <CAMvtBPNrWQGphKyQixJQCPbRLDODzTx_Ct4YQ4zd8ia1gnHVGQ@mail.gmail.com>
References: <1750999994.6369.1509476065042.JavaMail.zimbra@li.nux.ro> <CAMvtBPMCKB1KJeYw4a-tY_XHNYj+SR08v_1SEz5VsYQpzhyyuQ@mail.gmail.com> <270992313.1239.1509970237965.JavaMail.zimbra@li.nux.ro> <CAD97ioeHboORsFXbdLjQZ6rCH01Esbw+3+j4Q6tLamRuSafCZA@mail.gmail.com> <1778806379.4501.1510211582337@ox.pcextreme.nl> <CAMvtBPNrWQGphKyQixJQCPbRLDODzTx_Ct4YQ4zd8ia1gnHVGQ@mail.gmail.com>
Subject: Re: HTTPS LB and x-forwarded-for
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Thread-Topic: HTTPS LB and x-forwarded-for
Thread-Index: FQTS54JeuHn0NFcKQt6q2zXwub6bNA==
archived-at: Thu, 09 Nov 2017 19:05:29 -0000

Wido,

Excellent suggestion with the "transparent proxy", I was not aware of that.
I think that would be a great idea and wouldn't require too many modificati=
ons, especially as Haproxy comes already with the VR.

To Paul:
- imho the LB solution ACS ships now is a bit handicaped since you do not k=
now the remote host ip. You're flying blind unless you use google analytics=
 (and these things have gotten more and more aggressively filtered by adblo=
cks).
Enhancing Haproxy as Wido suggested would go a long way, it wouldn't break =
existing functionality and would also keep SSL processing off the VR.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Andrija Panic" <andrija.panic@gmail.com>
> To: "users" <users@cloudstack.apache.org>
> Cc: "Khosrow Moossavi" <kmoossavi@cloudops.com>, "Will Stevens" <wstevens=
@cloudops.com>, "dev"
> <dev@cloudstack.apache.org>, "Pierre-Luc Dion" <pdion@cloudops.com>
> Sent: Thursday, 9 November, 2017 13:10:58
> Subject: Re: HTTPS LB and x-forwarded-for

> Wido,
>=20
> backend servers are not Linux only, for example we have a ton of Windows
> customers, some WEB solutions / IIS etc...
>=20
> @all - If we try to please/solve everyone's proxying solution/requirement=
 -
> this is impossible IMHO - I'm thinking more about some "do it as you like=
"
> solution, to let customer write his own haproxy config and upoad it (for
> example, or something better?).
>=20
> We can support newer version of haproxy (1.5+) which also implement
> "transarent proxy" (integrate with kernel so to speak)  to allow TCP-leve=
l
> connections to backend (TCP mode, not HTTP mode) but to still "preserve"
> remote IP by faking it (fake soruce IP =3D transarent proxy).
>=20
> For the rest of configuration options,  I would leave it to the customer
> how he/she wants to configure rest of haproxy configuration, inlcuding
> custom checks, etc. Haproxy configuration is never-ending story, and we
> probably should allow custom sripts/configuration instead of trying to
> provide GUI/API way to configure everything (which is impossible...)
>=20
> Just my 2 cents...
>=20
> On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh.nl> wrote:
>=20
>>
>> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <pdion@cloudops.co=
m
>> >:
>> >
>> >
>> > Same challenge here too!
>> >
>> > Let's look at improving Load-balancing offering from cloudstack, I gue=
st
>> we
>> > should do a feature spec draft soon..,  from my perspective, doing SSL
>> > offload on the VR could be problematic if the VR spec if too small, an=
d
>> the
>> > default spec of the VR being 1vcpu@256MB, considering it can be the
>> router
>> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What wo=
uld
>> > be your thought about this ?
>> >
>> > I'd be curious to have a LB offering in ACS where it would deploy a
>> > redundant traefik[1] beside the VR for doing http and https
>> Load-balancing.
>> > I think it would also be useful if the API of that traefik instance wo=
uld
>> > be available from within the VPC or LBnetwork so is API would be
>> accessible
>> > to other apps orchestration tools such as  kubernetes or rancher.
>> >
>> > traefik or not, here is what I think is needed by cloudstack in the LB
>> > improvement:
>> >
>> > - support http, https (X-Forwarded-For)
>>
>> HAProxy also supports the PROXY protocol towards the backends. Apache
>> 2.4.22 supports this natively and Varnish for example can also talk PROX=
Y.
>>
>> It adds a littlebit of metadata to the connection so that the backend
>> knows the original IP the connection came from for example:
>> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
>>
>> Wido
>>
>> > - basic persistence tuning (API already exist)
>> > - better backend monitoring, currently only a tcp connect validate if =
the
>> > webserver is up.
>> > - ssl offload
>> > - metric collection, more stats, maybe just export the tool status pag=
e
>> to
>> > the private network.
>> > - Container world support, right now if you have Rancher or kubernetes
>> > cluster, you need to deploy your own LB solution behing mostlikely a
>> static
>> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher co=
uld
>> > reuse this instance and managed it to properly do LB between container=
s.
>> >
>> >
>> > What would be your prefered LB tool:
>> > haproxy, traefik or nginx?
>> >
>> > CloudStack already have to code to handle SSL certs per projects and
>> > accounts if not mistaking because that code was added to support
>> NetScaler
>> > as Load-balancer in the past. so one less thing to think about :-)
>> >
>> >
>> > [1] https://traefik.io/
>> >
>> >
>> > PL,
>> >
>> >
>> >
>> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nux@li.nux.ro> wrote:
>> >
>> > > Thanks Andrija,
>> > >
>> > > LB outside of the VR sounds like a good idea. An appliance based on,
>> say
>> > > cloud-init + ansible and so on could do the trick; alas it'd need to=
 be
>> > > outside ACS.
>> > > I guess as users we could maybe come up with a spec for an
>> improvement, at
>> > > least we'd have something the devs could look at whenever it is
>> possible.
>> > >
>> > > Regards,
>> > > Lucian
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> > >
>> > > ----- Original Message -----
>> > > > From: "Andrija Panic" <andrija.panic@gmail.com>
>> > > > To: "dev" <dev@cloudstack.apache.org>
>> > > > Cc: "users" <users@cloudstack.apache.org>
>> > > > Sent: Thursday, 2 November, 2017 23:21:37
>> > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > >
>> > > > We used to make some special stuff for one of the clients, where a=
ll
>> LB
>> > > > configuration work is done from outside of the ACS, i.e. python
>> script to
>> > > > feed/configure VR - install latest haproxy 1.5.x for transparent
>> proxy,
>> > > > since client insisted on SSL termination done on backend web SSL
>> > > servers....
>> > > > Not good idea, that is all I can say (custom configuration thing) =
-
>> but
>> > > the
>> > > > LB setup is actually good - transparent mode haproxy, works on TCP
>> level,
>> > > > so you can see "real client IP" on the backend servers (which must
>> use VR
>> > > > as the default gtw, as per default, so the whole setup works
>> properly).
>> > > >
>> > > > I'm still looking forward to see some special support of LB inside
>> VR via
>> > > > ACS - proper LB setup inside VR via GUI/API -  i.e. to enable LB
>> > > > provisioning SCRIPT (bash, or whatever),  where all needed
>> > > > install+configure can be done from client side  - otherwise coveri=
ng
>> all
>> > > > user cases, with proper HTTP checks and similar....is impossible t=
o
>> do
>> > > > IMHO.
>> > > >
>> > > > Some other clients, actually have internal FW appliance (i.e.
>> multihomed
>> > > > VM, acting as gtw for all VMs in all networks), and haproxy instal=
ed
>> on
>> > > > this device (with NAT configured from VR to this internal FW/VM, s=
o
>> > > remote
>> > > > IP can be seen properly) - this setup is fully under customer
>> control,
>> > > and
>> > > > can provide any kind of special haproxy config...
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On 31 October 2017 at 19:54, Nux! <nux@li.nux.ro> wrote:
>> > > >
>> > > >> Hello,
>> > > >>
>> > > >> Of the people running an LB (VR) with https backends, how do you
>> deal
>> > > with
>> > > >> the lack of x-forwarded-for since for port 443 there's just simpl=
e
>> TCP
>> > > >> balancing?
>> > > >>
>> > > >> Has anyone thought of terminating SSL in the VR instead? Ideas?
>> > > >>
>> > > >> Cheers
>> > > >>
>> > > >> --
>> > > >> Sent from the Delta quadrant using Borg technology!
>> > > >>
>> > > >> Nux!
>> > > >> www.nux.ro
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Andrija Pani=C4=87
>> > >
>>
>=20
>=20
>=20
> --
>=20
> Andrija Pani=C4=87