cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Luc Dion <pd...@cloudops.com>
Subject Re: HTTPS LB and x-forwarded-for
Date Mon, 13 Nov 2017 14:14:48 GMT
Hi,

This is looking quite promising, I have a colleague that tested that last
Friday, look like the proxy proto v1 work out of the box with Nginx, but
would need an extra package for Apache 2.4 ?
Otherwise, it seems to be a good way to do  https LB without complicated
configuration and huge changes in CloudStack.



On Fri, Nov 10, 2017 at 10:36 AM, Nux! <nux@li.nux.ro> wrote:

> Pierre-Luc,
>
> Haproxy docs say it should work for any kind of traffic as long as both
> ends are PROXY-aware and it look like a majority of software is.
> So, in short, yes.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Pierre-Luc Dion" <pdion@cloudops.com>
> > To: "Wido den Hollander" <wido@widodh.nl>
> > Cc: "dev" <dev@cloudstack.apache.org>, "Khosrow Moossavi" <
> kmoossavi@cloudops.com>, "Will Stevens"
> > <wstevens@cloudops.com>, "Nux!" <nux@li.nux.ro>, "users" <
> users@cloudstack.apache.org>
> > Sent: Friday, 10 November, 2017 15:32:38
> > Subject: Re: HTTPS LB and x-forwarded-for
>
> > Hi Wido, do you know if this would work for https traffic too?
> >
> > Le 10 nov. 2017 09 h 35, "Wido den Hollander" <wido@widodh.nl> a écrit :
> >
> >>
> >> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion <
> pdion@cloudops.com
> >> >:
> >> >
> >> >
> >> > I kind of like the proxy backend type, ill check on our end if that
> would
> >> > work but definitely a simple and efficient approach!
> >> >
> >>
> >> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
> >>
> >> Apache HTTPd supports PROXY since 2.4.28:
> https://httpd.apache.org/docs/
> >> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
> >>
> >> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
> >>
> >> Wido
> >>
> >> >
> >> >
> >> > Le 10 nov. 2017 01 h 44, "Wido den Hollander" <wido@widodh.nl> a
> écrit :
> >> >
> >> > >
> >> > > > Op 9 november 2017 om 19:59 schreef Nux! <nux@li.nux.ro>:
> >> > > >
> >> > > >
> >> > > > Wido,
> >> > > >
> >> > > > Excellent suggestion with the "transparent proxy", I was not
> aware of
> >> > > that.
> >> > > > I think that would be a great idea and wouldn't require too many
> >> > > modifications, especially as Haproxy comes already with the VR.
> >> > > >
> >> > >
> >> > > It's indeed just a matter of a HAProxy config setting. We could
> make it
> >> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> >> example.
> >> > >
> >> > > That way your problem would be solved.
> >> > >
> >> > > Wido
> >> > >
> >> > > > To Paul:
> >> > > > - imho the LB solution ACS ships now is a bit handicaped since
> you do
> >> > > not know the remote host ip. You're flying blind unless you use
> google
> >> > > analytics (and these things have gotten more and more aggressively
> >> filtered
> >> > > by adblocks).
> >> > > > Enhancing Haproxy as Wido suggested would go a long way, it
> wouldn't
> >> > > break existing functionality and would also keep SSL processing off
> >> the VR.
> >> > > >
> >> > > > --
> >> > > > Sent from the Delta quadrant using Borg technology!
> >> > > >
> >> > > > Nux!
> >> > > > www.nux.ro
> >> > > >
> >> > > > ----- Original Message -----
> >> > > > > From: "Andrija Panic" <andrija.panic@gmail.com>
> >> > > > > To: "users" <users@cloudstack.apache.org>
> >> > > > > Cc: "Khosrow Moossavi" <kmoossavi@cloudops.com>, "Will
> Stevens" <
> >> > > wstevens@cloudops.com>, "dev"
> >> > > > > <dev@cloudstack.apache.org>, "Pierre-Luc Dion" <
> pdion@cloudops.com
> >> >
> >> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> >> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > > >
> >> > > > > Wido,
> >> > > > >
> >> > > > > backend servers are not Linux only, for example we have
a ton of
> >> > > Windows
> >> > > > > customers, some WEB solutions / IIS etc...
> >> > > > >
> >> > > > > @all - If we try to please/solve everyone's proxying
> >> > > solution/requirement -
> >> > > > > this is impossible IMHO - I'm thinking more about some "do
it as
> >> you
> >> > > like"
> >> > > > > solution, to let customer write his own haproxy config and
> upoad it
> >> > > (for
> >> > > > > example, or something better?).
> >> > > > >
> >> > > > > We can support newer version of haproxy (1.5+) which also
> implement
> >> > > > > "transarent proxy" (integrate with kernel so to speak) 
to allow
> >> > > TCP-level
> >> > > > > connections to backend (TCP mode, not HTTP mode) but to
still
> >> > > "preserve"
> >> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> >> > > > >
> >> > > > > For the rest of configuration options,  I would leave it
to the
> >> > > customer
> >> > > > > how he/she wants to configure rest of haproxy configuration,
> >> inlcuding
> >> > > > > custom checks, etc. Haproxy configuration is never-ending
story,
> >> and we
> >> > > > > probably should allow custom sripts/configuration instead
of
> >> trying to
> >> > > > > provide GUI/API way to configure everything (which is
> >> impossible...)
> >> > > > >
> >> > > > > Just my 2 cents...
> >> > > > >
> >> > > > > On 9 November 2017 at 08:13, Wido den Hollander <wido@widodh.nl
> >
> >> > > wrote:
> >> > > > >
> >> > > > >>
> >> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc
Dion <
> >> > > pdion@cloudops.com
> >> > > > >> >:
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > Same challenge here too!
> >> > > > >> >
> >> > > > >> > Let's look at improving Load-balancing offering
from
> >> cloudstack, I
> >> > > guest
> >> > > > >> we
> >> > > > >> > should do a feature spec draft soon..,  from my
perspective,
> >> doing
> >> > > SSL
> >> > > > >> > offload on the VR could be problematic if the VR
spec if too
> >> small,
> >> > > and
> >> > > > >> the
> >> > > > >> > default spec of the VR being 1vcpu@256MB, considering
it can
> >> be the
> >> > > > >> router
> >> > > > >> > of a VPC, doing VPN termination, adding HTTPS 
is a bit
> ish...
> >> What
> >> > > would
> >> > > > >> > be your thought about this ?
> >> > > > >> >
> >> > > > >> > I'd be curious to have a LB offering in ACS where
it would
> >> deploy a
> >> > > > >> > redundant traefik[1] beside the VR for doing http
and https
> >> > > > >> Load-balancing.
> >> > > > >> > I think it would also be useful if the API of that
traefik
> >> instance
> >> > > would
> >> > > > >> > be available from within the VPC or LBnetwork so
is API
> would be
> >> > > > >> accessible
> >> > > > >> > to other apps orchestration tools such as  kubernetes
or
> >> rancher.
> >> > > > >> >
> >> > > > >> > traefik or not, here is what I think is needed
by cloudstack
> in
> >> the
> >> > > LB
> >> > > > >> > improvement:
> >> > > > >> >
> >> > > > >> > - support http, https (X-Forwarded-For)
> >> > > > >>
> >> > > > >> HAProxy also supports the PROXY protocol towards the
backends.
> >> Apache
> >> > > > >> 2.4.22 supports this natively and Varnish for example
can also
> >> talk
> >> > > PROXY.
> >> > > > >>
> >> > > > >> It adds a littlebit of metadata to the connection so
that the
> >> backend
> >> > > > >> knows the original IP the connection came from for example:
> >> > > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> >> > > > >>
> >> > > > >> Wido
> >> > > > >>
> >> > > > >> > - basic persistence tuning (API already exist)
> >> > > > >> > - better backend monitoring, currently only a tcp
connect
> >> validate
> >> > > if the
> >> > > > >> > webserver is up.
> >> > > > >> > - ssl offload
> >> > > > >> > - metric collection, more stats, maybe just export
the tool
> >> status
> >> > > page
> >> > > > >> to
> >> > > > >> > the private network.
> >> > > > >> > - Container world support, right now if you have
Rancher or
> >> > > kubernetes
> >> > > > >> > cluster, you need to deploy your own LB solution
behing
> >> mostlikely a
> >> > > > >> static
> >> > > > >> > nat., If cloudstack would deploy a traefik instance,
Kub or
> >> Rancher
> >> > > could
> >> > > > >> > reuse this instance and managed it to properly
do LB between
> >> > > containers.
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > What would be your prefered LB tool:
> >> > > > >> > haproxy, traefik or nginx?
> >> > > > >> >
> >> > > > >> > CloudStack already have to code to handle SSL certs
per
> >> projects and
> >> > > > >> > accounts if not mistaking because that code was
added to
> support
> >> > > > >> NetScaler
> >> > > > >> > as Load-balancer in the past. so one less thing
to think
> about
> >> :-)
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > [1] https://traefik.io/
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > PL,
> >> > > > >> >
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! <nux@li.nux.ro>
wrote:
> >> > > > >> >
> >> > > > >> > > Thanks Andrija,
> >> > > > >> > >
> >> > > > >> > > LB outside of the VR sounds like a good idea.
An appliance
> >> based
> >> > > on,
> >> > > > >> say
> >> > > > >> > > cloud-init + ansible and so on could do the
trick; alas
> it'd
> >> need
> >> > > to be
> >> > > > >> > > outside ACS.
> >> > > > >> > > I guess as users we could maybe come up with
a spec for an
> >> > > > >> improvement, at
> >> > > > >> > > least we'd have something the devs could look
at whenever
> it
> >> is
> >> > > > >> possible.
> >> > > > >> > >
> >> > > > >> > > Regards,
> >> > > > >> > > Lucian
> >> > > > >> > >
> >> > > > >> > > --
> >> > > > >> > > Sent from the Delta quadrant using Borg technology!
> >> > > > >> > >
> >> > > > >> > > Nux!
> >> > > > >> > > www.nux.ro
> >> > > > >> > >
> >> > > > >> > > ----- Original Message -----
> >> > > > >> > > > From: "Andrija Panic" <andrija.panic@gmail.com>
> >> > > > >> > > > To: "dev" <dev@cloudstack.apache.org>
> >> > > > >> > > > Cc: "users" <users@cloudstack.apache.org>
> >> > > > >> > > > Sent: Thursday, 2 November, 2017 23:21:37
> >> > > > >> > > > Subject: Re: HTTPS LB and x-forwarded-for
> >> > > > >> > >
> >> > > > >> > > > We used to make some special stuff for
one of the
> clients,
> >> > > where all
> >> > > > >> LB
> >> > > > >> > > > configuration work is done from outside
of the ACS, i.e.
> >> python
> >> > > > >> script to
> >> > > > >> > > > feed/configure VR - install latest haproxy
1.5.x for
> >> transparent
> >> > > > >> proxy,
> >> > > > >> > > > since client insisted on SSL termination
done on backend
> >> web SSL
> >> > > > >> > > servers....
> >> > > > >> > > > Not good idea, that is all I can say
(custom
> configuration
> >> > > thing) -
> >> > > > >> but
> >> > > > >> > > the
> >> > > > >> > > > LB setup is actually good - transparent
mode haproxy,
> works
> >> on
> >> > > TCP
> >> > > > >> level,
> >> > > > >> > > > so you can see "real client IP" on the
backend servers
> >> (which
> >> > > must
> >> > > > >> use VR
> >> > > > >> > > > as the default gtw, as per default, so
the whole setup
> works
> >> > > > >> properly).
> >> > > > >> > > >
> >> > > > >> > > > I'm still looking forward to see some
special support of
> LB
> >> > > inside
> >> > > > >> VR via
> >> > > > >> > > > ACS - proper LB setup inside VR via GUI/API
-  i.e. to
> >> enable LB
> >> > > > >> > > > provisioning SCRIPT (bash, or whatever),
 where all
> needed
> >> > > > >> > > > install+configure can be done from client
side  -
> otherwise
> >> > > covering
> >> > > > >> all
> >> > > > >> > > > user cases, with proper HTTP checks and
similar....is
> >> > > impossible to
> >> > > > >> do
> >> > > > >> > > > IMHO.
> >> > > > >> > > >
> >> > > > >> > > > Some other clients, actually have internal
FW appliance
> >> (i.e.
> >> > > > >> multihomed
> >> > > > >> > > > VM, acting as gtw for all VMs in all
networks), and
> haproxy
> >> > > instaled
> >> > > > >> on
> >> > > > >> > > > this device (with NAT configured from
VR to this internal
> >> > > FW/VM, so
> >> > > > >> > > remote
> >> > > > >> > > > IP can be seen properly) - this setup
is fully under
> >> customer
> >> > > > >> control,
> >> > > > >> > > and
> >> > > > >> > > > can provide any kind of special haproxy
config...
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > > On 31 October 2017 at 19:54, Nux! <nux@li.nux.ro>
wrote:
> >> > > > >> > > >
> >> > > > >> > > >> Hello,
> >> > > > >> > > >>
> >> > > > >> > > >> Of the people running an LB (VR)
with https backends,
> how
> >> do
> >> > > you
> >> > > > >> deal
> >> > > > >> > > with
> >> > > > >> > > >> the lack of x-forwarded-for since
for port 443 there's
> just
> >> > > simple
> >> > > > >> TCP
> >> > > > >> > > >> balancing?
> >> > > > >> > > >>
> >> > > > >> > > >> Has anyone thought of terminating
SSL in the VR instead?
> >> Ideas?
> >> > > > >> > > >>
> >> > > > >> > > >> Cheers
> >> > > > >> > > >>
> >> > > > >> > > >> --
> >> > > > >> > > >> Sent from the Delta quadrant using
Borg technology!
> >> > > > >> > > >>
> >> > > > >> > > >> Nux!
> >> > > > >> > > >> www.nux.ro
> >> > > > >> > > >>
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > > > --
> >> > > > >> > > >
> >> > > > >> > > > Andrija Panić
> >> > > > >> > >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > > Andrija Panić
> >> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message