mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elouan Keryell-Even <elouan.kery...@gmail.com>
Subject Run Mesos without master being able to open connections to slaves
Date Mon, 25 Apr 2016 15:35:28 GMT
Hi all,

I work on a R&D project where I need to make two computing clusters
collaborate together (an industrial datacenter & a cluster running in the
cloud). One of the two clusters acts as the "main" cluster, the other as
the "secondary" one. The idea is to test "bursting", i.e. when the main
cluster is full it will send jobs to the secondary cluster, so that it can
overcome the current load peak.

Now this is where it gets complex: as it is a small R&D project which
interacts with big industrial infrastructures, we face some strict network
restrictions (security oblige). We were able to have the authorization to
open an outgoing SSH tunnel (from the industrial data center to the cloud),
but not an ingoing one. And, of course, we have not the authorization to
work around this restriction by using an outgoing tunnel in reverse mode.

I guess this could be overcome if there was a way to make it work with
1-way communications, i.e. one of the two sides only (master or slave)
could take care of initiating all the connections. I think the slave
absolutely needs to be able to open a connection to the master, since it is
him who initiate his own registration to the cluster. But on the other hand
I was not 100% sure if the master also needed to initiate connections
(maybe to cancel tasks).

So I tried this configuration:

Datacenter ===============> Cloud

^^^^^^^      ^^^^^^^^^^^^^^^^^^     ^^^^^^^
slaves       1-way SSH tunnel      master

The slave can reach the master in the beginning, but it doesn't work for
very long because afterwards because from the master point of view it keeps
switching between connected/disconnected state. I think what happens is
that the slave successfully reach the master for registration, but then
when the master tries to check if it is still alive (which happens
periodically I guess), it can't reach it (unable to open connection) and
thinks it is disconnected. Then the slave registers again, etc etc...

I tried to look for a similar problem on the web, and I think I found
evidences that the master needs the ability to open connections back to the
slaves:

https://mail-archives.apache.org/mod_mbox/mesos-user/201412.mbox/%3CCA+8RcoRxMR2nk-SA9iPYK_uVUYr8k7xEh_AbL69R0jnb3uLJMA@mail.gmail.com%3E

http://stackoverflow.com/a/32275220/3037171

http://stackoverflow.com/a/24559617/3037171


However, the latest link dates back to ~September 2015, and I personnally
use Mesos 0.22.1 which dates back to ~May 2015. So I was wondering if this
particular network behavior could be overcome with the latest versions, but
I quickly read through the changelogs and didn't notice anything relative
to that. I also dug into the code for several hours, but I found it hard to
understand precisely how the communication architecture of Mesos works.


So I'd be glad to have some insight from you guys about if it is possible,
in one way or another, to make Mesos work without the Master being able to
initiate connections to slaves. I just need to be 100% sure there isn't any
workaround before going back to my boss :)


Thank you very much for your attention!

Elouan

Mime
View raw message