mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Sallou <olivier.sal...@irisa.fr>
Subject Re: Fail to get CNI with unified containerizer, job remains stuck on staging
Date Wed, 24 Aug 2016 06:23:07 GMT


On 08/23/2016 06:13 PM, Jie Yu wrote:
> The DNS related logging means that the weave plugin does not return DNS
> information, the agent uses the host resolv.conf for the container. So I
> think is irrelevant to your problem.
>
> Mesos requires that executor can talk to agent. Can you see if there is a
> route from 10.32.0.1 to the agent IP?
How can I check this as task does not start ? I have exposed weave
network on host:

osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave expose
10.32.0.2
osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms

 And why is it blocking?

I am on a single host environement, so agent is on 127.0.0.1.

Olivier
>
> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <olivier.sallou@irisa.fr>
> wrote:
>
>> HI,
>>
>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
>>
>> Weave works nicely with the Docker containerizer.
>>
>> When I try to launch a task via my framework with unified containerizer,
>> the job remains waiting forever (no RUNNING message). I can see however
>> that weave cni allocated an IP address to Mesos.
>>
>> I tried with a simple mesos-execute test.
>>
>> Example with a mesos-execute with no CNI, everything is OK
>>
>>
>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>> --name=test0  I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: 1.0.0
>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
>> master@127.0.0.1:5050
>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>> Received status update TASK_RUNNING for task 'test0'
>>   source: SOURCE_EXECUTOR
>> Received status update TASK_FINISHED for task 'test0'
>>   message: 'Command exited with status 0'
>>
>>
>> Sample example specifying the weave network
>>
>>
>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>> --name=test0   --networks=weave
>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
>> master@127.0.0.1:5050
>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
>>
>> mesos-slave logs:
>>
>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
>> '/proc/28869/ns/net' to
>> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
>> '10.32.0.1/12' from CNI network 'weave' for container
>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
>> '/etc/resolv.conf'
>>
>> There are no other logs until I kill the job.
>> We can see that Mesos container got an IP but it seems to block on DNS,
>>
>> Thanks for hints....
>>
>> --
>>
>> gpg key id: 4096R/326D8438  (keyring.debian.org)
>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>>
>>
>>

-- 
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95

gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438


Mime
View raw message