mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Jordan <elias.k.jor...@gmail.com>
Subject Re: Set LIBPROCESS_IP for frameworks launched with marathon
Date Tue, 07 Jun 2016 14:43:28 GMT
Currently I have it configured to use host networking

Thanks
Eli

> On 7 Jun 2016, at 11:25, Radoslaw Gruchalski <radek@gruchalski.com> wrote:
> 
> Yes, because that runs in host network. This leads to a question: your docker task, is
it bridge or host network.
> 
> -- 
> Best regards,
> Rad
> 
> 
> 
> 
> On Tue, Jun 7, 2016 at 3:21 AM +0200, "Eli Jordan" <elias.k.jordan@gmail.com> wrote:
> 
>> It's important to note that if you run a task with the command executor (I.e. Not
using docker) LIBPROCESS_IP is defined, along with several other variables that are not defined
in docker.
>> 
>> Thanks
>> Eli
>> 
>>> On 7 Jun 2016, at 10:05, Radoslaw Gruchalski <radek@gruchalski.com> wrote:
>>> 
>>> I think the problem is that it is not known which agent the task is running on
until the task i in the running state.
>>> Hence the master can’t pass that as an env variable to the task.
>>> However, I see your point. There is an agent host name avaialble in the task
as $HOST. Maybe it would be a good idea if Mesos was setting env variables named AGENT_IP_0,
AGENT_IP_1 and so on for every IP interface on the agent, maybe AGENT_BIND_IP if bind IP is
different than 0.0.0.0. OTOH, I can see how this could be considered as some security issue.
I am not sure what the implications could be.
>>> 
>>> Anybody else care to comment?
>>> – 
>>> Best regards,
>>> 
>>> Radek Gruchalski
>>> 
>>> radek@gruchalski.com
>>> de.linkedin.com/in/radgruchalski
>>> 
>>> Confidentiality:
>>> This communication is intended for the above-named person and may be confidential
and/or legally privileged.
>>> If it has come to you in error you must take no action based on it, nor must
you copy or show it to anyone; please delete/destroy and inform the sender immediately.
>>> 
>>>> On June 7, 2016 at 1:42:46 AM, Eli Jordan (elias.k.jordan@gmail.com) wrote:
>>>> 
>>>> Thanks Radoslaw. I'm not really set on using host names, I just want a reliable
way to start the framework. For the meantime I have gone with a solution similar to what you
suggested. We use /etc/default/mesos file to configure mesos, and it has the ip defined, so
I just mounted that in the container and read the ip.
>>>> 
>>>> I would like to avoid having a dependency on the file system of the  agents
though. I'm not sure why I can't have the docket executor set the LIBPROCESS_IP variable in
the same way the command executor does.
>>>> 
>>>> Thanks
>>>> Eli
>>>> 
>>>> On 6 Jun 2016, at 21:44, Radoslaw Gruchalski <radek@gruchalski.com>
wrote:
>>>> 
>>>>> Out of curiosity. Why are you insisting on using host names?
>>>>> Say you have 1 master and 2 agents with these IPs:
>>>>> 
>>>>> - mesos-master-0: 10.100.1.10
>>>>> - mesos-agent-0: 10.100.1.11
>>>>> - mesos-agent-1: 10.100.1.12
>>>>> 
>>>>> Your problem is that you have no way to obtain an IP address of the agent
in the container. Correct?
>>>>> One way to overcome this problem is to create a shell file, say in /etc/mesos-agent.sh,
with contents like:
>>>>> 
>>>>> ...
>>>>> AGENT_IP=10.100.1.11
>>>>> …
>>>>> 
>>>>> If you’re using Marathon, you can copy that file to the sandbox using
docker volumes:
>>>>> 
>>>>>             {
>>>>>                 "containerPath": “/etc/mesos-agent.sh",
>>>>>                 "hostPath": "/etc/mesos-agent.sh",
>>>>>                 "mode": "RO"
>>>>>             }
>>>>> 
>>>>> You can now source that in the container to set the LIBPROCESS_ADVERTISE_IP.
>>>>> Other applications simply use the mesos-agent-X host name. That’s without
mesos-dns.
>>>>> Things are easier with mesos-dns or consul service catalog (I prefer
the latter).
>>>>> – 
>>>>> Best regards,
>>>>> 
>>>>> Radek Gruchalski
>>>>> 
>>>>> radek@gruchalski.com
>>>>> de.linkedin.com/in/radgruchalski
>>>>> 
>>>>> Confidentiality:
>>>>> This communication is intended for the above-named person and may be
confidential and/or legally privileged.
>>>>> If it has come to you in error you must take no action based on it, nor
must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
>>>>> 
>>>>>> On June 6, 2016 at 1:16:07 PM, Eli Jordan (elias.k.jordan@gmail.com)
wrote:
>>>>>> 
>>>>>> The issue refers to LIBPROCESS_IP not LIBPROCESS_HOST. I haven’t
been able to find the LIBPROCESS_HOST variable documented anywhere.
>>>>>> 
>>>>>> My understanding is that the scheduler uses LIBPROCESS_IP to determine
which network interface to bind to, and also which ip to advertise to the master, so that
the master can send offers. There is also another variable LIBPROCESS_ADVERTISE_IP. If this
is defined then LIBPROCESS_IP is used to determine which network interface to bind to, and
LIBPROCESS_ADVERTISE_IP is used to determine which ip to advertise to the master.
>>>>>> 
>>>>>> It would be great if there was a LIBPROCESS_ADVERTISE_HOST variable,
then I could just use the $HOST variable to define this.
>>>>>> 
>>>>>>> On 5 Jun 2016, at 10:41 pm, Sivaram Kannan <sivaramsk@gmail.com>
wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> I have been using this way from 0.23.0 to the 0.28.0. This has
been surely working (although for a different framework). Inside the docker container can
you see the $HOST variable defined??
>>>>>>> 
>>>>>>> The ticket you referred says that the apps definition needs to
define LIBPROCESS_HOST=$HOST to be make the framework take the proper IP - you are describing
a different problem.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> ./Siva.
>>>>>>> 
>>>>>>>> On Sun, Jun 5, 2016 at 4:30 AM, Eli Jordan <elias.k.jordan@gmail.com>
wrote:
>>>>>>>> I found this issue on the mesos jira that describes the exact
issue I am hitting.
>>>>>>>> 
>>>>>>>> https://issues.apache.org/jira/browse/MESOS-3740
>>>>>>>> 
>>>>>>>> It doesn't appear to be resolved. 
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Eli
>>>>>>>> 
>>>>>>>> On 5 Jun 2016, at 16:46, Eli Jordan <elias.k.jordan@gmail.com>
wrote:
>>>>>>>> 
>>>>>>>>> Hmmm… that doesn’t seem to work for me. What version
of mesos does this work in? I am running 0.27.1. 
>>>>>>>>> 
>>>>>>>>> When using this approach, I still get the following error
when the kafka mess framework is starting up.
>>>>>>>>> 
>>>>>>>>> "Scheduler driver bound to loopback interface! Cannot
communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable
to use a routable IP address.”
>>>>>>>>> 
>>>>>>>>> I tried setting LIBPROCESS_IP to ‘0.0.0.0’ and LIBPROCESS_ADVERTISE_IP=‘the
public ip’ and this works. But the host variations don’t seem to work. (i.e. set LIBPROCESS_IP=0.0.0.0
and LIBPROCESS_ADVERTISE_HOST=$HOST)
>>>>>>>>> 
>>>>>>>>> It seems lib process doesn’t support using host names.
>>>>>>>>> 
>>>>>>>>> I think I might have to run the framework outside of
docker, but I would really like to avoid this. 
>>>>>>>>> 
>>>>>>>>> This problem would be solved if the docker executor was
able to set the same environment variables as the command executor. Is there a way to make
this happen?
>>>>>>>>> 
>>>>>>>>> I saw that mesos can be extended with a Hook ‘module’
to set extra environment variables in docker containers. This might be a solution, but seems
over wrought for a simple problem.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On 5 Jun 2016, at 12:50 am, Sivaram Kannan <sivaramsk@gmail.com>
wrote:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> Can you try adding && after the LIBPROCESS_HOST
variable and the actual command. We have been using this for sometime now.
>>>>>>>>>> 
>>>>>>>>>> "cmd": "LIBPROCESS_HOST=$HOST && ./kafka-mesos.sh
..
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> ./Siva.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jun 4, 2016 at 8:34 AM, Eli Jordan <elias.k.jordan@gmail.com>
wrote:
>>>>>>>>>>> Hi @haosdent
>>>>>>>>>>> 
>>>>>>>>>>> Based on my testing, this is not the case.
>>>>>>>>>>> 
>>>>>>>>>>> I ran a task (from marathon) without using a
docker container that just printed out all environment variables. i.e. while [ true ]; do
env; sleep 2; done
>>>>>>>>>>> 
>>>>>>>>>>> I then run a task that executed the same command
inside an alpine docker image.
>>>>>>>>>>> 
>>>>>>>>>>> When running without a docker image LIBPROCESS_IP
was defined along with many other variables. 
>>>>>>>>>>> 
>>>>>>>>>>> Sample output when running without docker (note
LIBPROCESS_IP) is defined
>>>>>>>>>>> 
>>>>>>>>>>> Registered executor on mesos-slave0
>>>>>>>>>>> Starting task plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e
>>>>>>>>>>> sh -c 'while [ true ]; do env; sleep 2; done'
>>>>>>>>>>> Forked command at 16571
>>>>>>>>>>> LIBPROCESS_IP=192.168.3.16
>>>>>>>>>>> MESOS_AGENT_ENDPOINT=192.168.3.16:5051
>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
>>>>>>>>>>> HOST=mesos-slave0
>>>>>>>>>>> SHELL=/bin/sh
>>>>>>>>>>> MESOS_DIRECTORY=/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/plain-test.5e5b00cc-2645-11e6-a3dd-080027aa149e/runs/c9b6ef86-b37d-4e3c-b1ca-bd680aed779f
>>>>>>>>>>> PORT0=31082
>>>>>>>>>>> PORT_10001=31082
>>>>>>>>>>> LC_ALL=en_US.UTF-8
>>>>>>>>>>> … more
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Sample output when running with docker (note
LIBPROCESS_IP is not defined)
>>>>>>>>>>> 
>>>>>>>>>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42"
--docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true"
--launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox"
--quiet="false" --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42"
--stop_timeout="0ns"
>>>>>>>>>>> --container="mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42"
--docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true"
--launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox"
--quiet="false" --sandbox_directory="/var/mesos/slaves/7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0/frameworks/aae929c7-24a5-4463-9ae0-bc7b044973c5-0000/executors/alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e/runs/f3a94ab4-dfff-4e97-b806-f1cc501ecf42"
--stop_timeout="0ns"
>>>>>>>>>>> Registered docker executor on mesos-slave0
>>>>>>>>>>> Starting task alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e
>>>>>>>>>>> HOSTNAME=984809b0b720
>>>>>>>>>>> SHLVL=1
>>>>>>>>>>> HOME=/root
>>>>>>>>>>> PORT=31295
>>>>>>>>>>> MESOS_CONTAINER_NAME=mesos-7ad17efe-0f9e-4703-9d2e-7fb9ee03f64c-S0.f3a94ab4-dfff-4e97-b806-f1cc501ecf42
>>>>>>>>>>> MARATHON_APP_ID=/alpine-test
>>>>>>>>>>> PORTS=31295
>>>>>>>>>>> PORT0=31295
>>>>>>>>>>> MARATHON_APP_DOCKER_IMAGE=alpine
>>>>>>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>>> MARATHON_APP_RESOURCE_DISK=0.0
>>>>>>>>>>> MARATHON_APP_RESOURCE_MEM=128.0
>>>>>>>>>>> HOST=mesos-slave0
>>>>>>>>>>> PORT_10000=31295
>>>>>>>>>>> MARATHON_APP_VERSION=2016-05-30T08:56:59.065Z
>>>>>>>>>>> MARATHON_APP_LABELS=
>>>>>>>>>>> PWD=/
>>>>>>>>>>> MESOS_TASK_ID=alpine-test.77d5a3d9-2644-11e6-a3dd-080027aa149e
>>>>>>>>>>> MARATHON_APP_RESOURCE_CPUS=1.0
>>>>>>>>>>> 
>>>>>>>>>>> Is there some other config I need to do for the
docker containerizer? Any help greatly appreciated.
>>>>>>>>>>> 
>>>>>>>>>>>> On 4 Jun 2016, at 7:28 pm, haosdent <haosdent@gmail.com>
wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi, @Jordan. I think not matter you use MesosContainerizer
or DockerContainerizer, LIBPROCESS_IP always would be set if you launch you Agent with `--ip`
flag.
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jun 3, 2016 at 8:23 PM, Eli Jordan
<elias.k.jordan@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The reason I need to set LIBPROCESS_IP
is because the slaves have 2 network interfaces, and the docker container is running in host
networking mode. So libmesos doesn’t know which IP to advertise. The hostnames of the slaves
are all resolvable.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have noticed that if I run a marathon
app that doesn’t use docker, e.g. while [ true ]; do env; sleep 2; done, that LIBPROCESS_IP
is defined in the environment. However, when running a docker image this variable is not defined.

>>>>>>>>>>>>> 
>>>>>>>>>>>>> Is there a way to have marathon pass
along all environment variables defined by mesos?
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Eli
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 4 Apr 2016, at 14:12, Eli Jordan <elias.k.jordan@gmail.com>
wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> @haosdent I’m not sure how this
works internally, but it seems the mess master needs to send requests to the framework for
resource offers, and therefore needs to know the external IP (i.e. the host IP)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> @craig w
>>>>>>>>>>>>>> Would I need to do this in the cmd
portion of the marathon JSON? I currently have the config below
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>     "container": ...,
>>>>>>>>>>>>>>     "id":"kafka-mesos-scheduler",
>>>>>>>>>>>>>>     "cpus": 0.5,
>>>>>>>>>>>>>>     "mem": 256,
>>>>>>>>>>>>>>     "ports": [9999],
>>>>>>>>>>>>>>     "cmd": "./kafka-mesos.sh scheduler
--master=mesos-master:5050 --zk=mesos-master:2181 --api=http://mesos-slave0:9999 --storage=zk:/kafka-mesos",
>>>>>>>>>>>>>>     "instances": 1,
>>>>>>>>>>>>>>     "constraints": [["hostname",
"LIKE", "mesos-slave0"]],
>>>>>>>>>>>>>>     "env": {
>>>>>>>>>>>>>>         "LIBPROCESS_IP": "192.168.3.16"
>>>>>>>>>>>>>>     }
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> @Chris Baker Currently we don’t
have mess-dns setup but if this works it would seem to be a nice solution. However, I did
try setting LIBPROCESS_IP to the slave hostname and it seems to produce a parse error. So
I think it needs to be an actual IP address.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I was hoping there would be a configuration
for the slave that would automatically populate this env variable when starting the docker
container. So I don’t need to complicate the marathon file, and can reuse it in different
clusters.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 4 Apr 2016, at 11:25 am, Chris
Baker <chris@galacticfog.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Alternatively, because the $HOST
username is indirect, which would require a runtime element to "export $LIBPROCESS_IP=$HOST",
another alternative is to fallback on Mesos-DNS, if that's part of the cluster deployment,
setting $LIBPROCESS_IP to the (a priori) Mesos-DNS entry corresponding to the service.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sun, Apr 3, 2016 at 5:06
PM craig w <codecraig@gmail.com> wrote:
>>>>>>>>>>>>>>>> Hi, marathon sets the HOST
env var. If it's not the ip address you can use getent with the value from HOST to figure
it out.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> >However, in order for
the frameworks to receive resource offers I need to set the LIBPROCESS_IP environment variable
to the hosts IP address for the docker container running the frameworks. 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, @Gmail. Could you provide
more details about this?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sun, Apr 3, 2016 at
10:40 PM, Rad Gruchalski <radek@gruchalski.com> wrote:
>>>>>>>>>>>>>>>>> Hi Gmail,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> AFAIK not. The only way
to do so is setting up the env variable as you do now.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>> Radek Gruchalski
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>
radek@gruchalski.com
>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>> de.linkedin.com/in/radgruchalski/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Confidentiality:
>>>>>>>>>>>>>>>>> This communication is
intended for the above-named person and may be confidential and/or legally privileged.
>>>>>>>>>>>>>>>>> If it has come to you
in error you must take no action based on it, nor must you copy or show it to anyone; please
delete/destroy and inform the sender immediately.
>>>>>>>>>>>>>>>>>> On Sunday, 3 April
2016 at 16:09, Gmail wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I'm pretty new to
mesos and marathon, and I'm running a couple of frameworks with marathon (Kafka and elastic
search). However, in order for the frameworks to receive resource offers I need to set the
LIBPROCESS_IP environment variable to the hosts IP address for the docker container running
the frameworks. Currently I am working around me this by using a constraint to hard wire the
slave that the framework gets launched on, so then I can put the slaves ip in the marathon
json file.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Obviously this is
not ideal. Is there a better way to define the host ip Inside the docker container?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Sent from my iPad
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Haosdent Huang
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Haosdent Huang
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> ever tried. ever failed. no matter.
>>>>>>>>>> try again. fail again. fail better.
>>>>>>>>>>         -- Samuel Beckett
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> ever tried. ever failed. no matter.
>>>>>>> try again. fail again. fail better.
>>>>>>>         -- Samuel Beckett

Mime
View raw message