mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Mesos/Marathon/HAProxy Logging
Date Tue, 25 Aug 2015 16:30:10 GMT
I have been playing with an application that is a very simple app: A
webservice running in Python. I've created a docker container, it runs in
the container, I setup marathon to run it, I use mesos-dns and ha proxy and
I can access the service just fine anywhere in the cluster.

First let me say this is VERY cool. The capabilities here awesome.

Now the challenge: the security guy in me wants to take good logs from my
app.  It was setup to do it's own logging through a custom module. I am
very happy with it.  I setup the app in the container to mount a volume
that's in my MapRFS via NFS so I can log directly to a clustered
filesystem. THis is awesome, I can read my logs in Apache Drill as they are
written!!!

However, the haproxy through me for a loop. Once I started running the app
in Marathon with a service port and routed around via haproxy, I realized
something:  I lost my source IPs in my logs?

Why?

Because once HAProxy takes over, it no longer needs to keep the source IP,
and instead the next hop only sees the previous connection IP.  From a
service discovery perspective it works great, but with this setup, I'd lose
the previous hop. Perhaps I manually add something in haproxy to add an
X-forwarded-for header, that would be nice, however, that only works for
http apps, what about other TCP apps that are not HTTP?

This is an interesting problem, because apps should have good logging,
security, performance, troubleshooting, and if I can't get the source IP it
could be a problem.

So, my question is this, anyone ran into this? How are you handling it?
Any brainstorms here we may be able to work off of?

One thing I thought was why are we using HAproxy? Couldn't the same HAProxy
script, actually put in forwarding rules in IPtables?  This sounds messy,
but could it work? Has anyone explored that? If the data was forwarded,
than it wouldn't lose the IP information (and timeouts wouldn't be a
concern either (I think I posted before on how long running TCP connections
can be closed down by HAProxy if they don't implement TCP Keep alives).

Other ideas?  This is interesting to me, and likely others.

Mime
View raw message