mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olivier Sallou <olivier.sal...@irisa.fr>
Subject Re: Fail to get CNI with unified containerizer, job remains stuck on staging
Date Wed, 24 Aug 2016 07:21:25 GMT


On 08/24/2016 08:23 AM, Olivier Sallou wrote:
>
> On 08/23/2016 06:13 PM, Jie Yu wrote:
>> The DNS related logging means that the weave plugin does not return DNS
>> information, the agent uses the host resolv.conf for the container. So I
>> think is irrelevant to your problem.
>>
>> Mesos requires that executor can talk to agent. Can you see if there is a
>> route from 10.32.0.1 to the agent IP?
> How can I check this as task does not start ? I have exposed weave
> network on host:
>
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo ./weave expose
> 10.32.0.2
> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ ping 10.32.0.2
> PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.
> 64 bytes from 10.32.0.2: icmp_seq=1 ttl=64 time=0.032 ms
> 64 bytes from 10.32.0.2: icmp_seq=2 ttl=64 time=0.029 ms
> 64 bytes from 10.32.0.2: icmp_seq=3 ttl=64 time=0.029 ms
> 64 bytes from 10.32.0.2: icmp_seq=4 ttl=64 time=0.031 ms
>
>  And why is it blocking?
>
> I am on a single host environement, so agent is on 127.0.0.1.
By the way, running a Docker container to use the weave CNI plugin works
fine, it gets it IP and container runs nicely.
>
> Olivier
>> On Tue, Aug 23, 2016 at 9:05 AM, Olivier Sallou <olivier.sallou@irisa.fr>
>> wrote:
>>
>>> HI,
>>>
>>> I have setup Mesos 1.0.0-2 to use CNI with Weave (1.6.1)
>>>
>>> Weave works nicely with the Docker containerizer.
>>>
>>> When I try to launch a task via my framework with unified containerizer,
>>> the job remains waiting forever (no RUNNING message). I can see however
>>> that weave cni allocated an IP address to Mesos.
>>>
>>> I tried with a simple mesos-execute test.
>>>
>>> Example with a mesos-execute with no CNI, everything is OK
>>>
>>>
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>>> --name=test0  I0823 17:56:50.067520 28815 scheduler.cpp:172] Version: 1.0.0
>>> I0823 17:56:50.068260 28822 scheduler.cpp:461] New master detected at
>>> master@127.0.0.1:5050
>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0005'
>>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>> Received status update TASK_RUNNING for task 'test0'
>>>   source: SOURCE_EXECUTOR
>>> Received status update TASK_FINISHED for task 'test0'
>>>   message: 'Command exited with status 0'
>>>
>>>
>>> Sample example specifying the weave network
>>>
>>>
>>> osallou@tifenn~/Development/NOSAVE/go-docker/weave $ sudo mesos-execute
>>> --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050
>>> --name=test0   --networks=weave
>>> I0823 17:57:15.845304 28856 scheduler.cpp:172] Version: 1.0.0
>>> I0823 17:57:15.846248 28857 scheduler.cpp:461] New master detected at
>>> master@127.0.0.1:5050
>>> Subscribed with ID 'b1925e13-76db-4225-a3dc-39ce65c79b3c-0006'
>>> Submitted task 'test0' to agent 'b1925e13-76db-4225-a3dc-39ce65c79b3c-S0'
>>> ==> REMAINS WAITING HERE, job is in STAGING in Mesos UI
>>>
>>> mesos-slave logs:
>>>
>>> I0823 17:57:15.873872 26522 cni.cpp:716] Bind mounted
>>> '/proc/28869/ns/net' to
>>> '/run/mesos/isolators/network/cni/4f91a5df-2e9a-4cfc-93f5-aa197646db09/ns'
>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>> I0823 17:57:16.257063 26519 cni.cpp:1109] Got assigned IPv4 address
>>> '10.32.0.1/12' from CNI network 'weave' for container
>>> 4f91a5df-2e9a-4cfc-93f5-aa197646db09
>>> I0823 17:57:16.257258 26525 cni.cpp:838] Unable to find DNS nameservers
>>> for container 4f91a5df-2e9a-4cfc-93f5-aa197646db09. Using host
>>> '/etc/resolv.conf'
>>>
>>> There are no other logs until I kill the job.
>>> We can see that Mesos container got an IP but it seems to block on DNS,
>>>
>>> Thanks for hints....
>>>
>>> --
>>>
>>> gpg key id: 4096R/326D8438  (keyring.debian.org)
>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>>>
>>>
>>>

-- 
Olivier Sallou
IRISA / University of Rennes 1
Campus de Beaulieu, 35000 RENNES - FRANCE
Tel: 02.99.84.71.95

gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438


Mime
View raw message