mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <>
Subject Re: Mesos/Marathon/HAProxy Logging
Date Tue, 25 Aug 2015 22:00:32 GMT
So I agree that is how it should be done, however the current
implementation on Mesos, requires me to manually code something like. In
addition, this is only for http traffic, not tcp... what happens when the
service running on Mesos isn't HTTP? I was hoping for some discussion
beyond just manually editing the ha proxy script to make it http and add
the headers...

On Tue, Aug 25, 2015 at 12:46 PM, Jeff Schroeder <
> wrote:

> This is the header that should be passed:
> Most of the modern internet routes through reverse proxies and this is how
> we log the actual source clients to solve similar auditing and compliance
> needs.
> On Tuesday, August 25, 2015, John Omernik <> wrote:
>> I have been playing with an application that is a very simple app: A
>> webservice running in Python. I've created a docker container, it runs in
>> the container, I setup marathon to run it, I use mesos-dns and ha proxy and
>> I can access the service just fine anywhere in the cluster.
>> First let me say this is VERY cool. The capabilities here awesome.
>> Now the challenge: the security guy in me wants to take good logs from my
>> app.  It was setup to do it's own logging through a custom module. I am
>> very happy with it.  I setup the app in the container to mount a volume
>> that's in my MapRFS via NFS so I can log directly to a clustered
>> filesystem. THis is awesome, I can read my logs in Apache Drill as they are
>> written!!!
>> However, the haproxy through me for a loop. Once I started running the
>> app in Marathon with a service port and routed around via haproxy, I
>> realized something:  I lost my source IPs in my logs?
>> Why?
>> Because once HAProxy takes over, it no longer needs to keep the source
>> IP, and instead the next hop only sees the previous connection IP.  From a
>> service discovery perspective it works great, but with this setup, I'd lose
>> the previous hop. Perhaps I manually add something in haproxy to add an
>> X-forwarded-for header, that would be nice, however, that only works for
>> http apps, what about other TCP apps that are not HTTP?
>> This is an interesting problem, because apps should have good logging,
>> security, performance, troubleshooting, and if I can't get the source IP it
>> could be a problem.
>> So, my question is this, anyone ran into this? How are you handling it?
>> Any brainstorms here we may be able to work off of?
>> One thing I thought was why are we using HAproxy? Couldn't the same
>> HAProxy script, actually put in forwarding rules in IPtables?  This sounds
>> messy, but could it work? Has anyone explored that? If the data was
>> forwarded, than it wouldn't lose the IP information (and timeouts wouldn't
>> be a concern either (I think I posted before on how long running TCP
>> connections can be closed down by HAProxy if they don't implement TCP Keep
>> alives).
>> Other ideas?  This is interesting to me, and likely others.
> --
> Text by Jeff, typos by iPhone

View raw message