mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ambuj Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MESOS-8065) Implicitly declining offers in ACCEPT call as the launch operation specified no tasks - for GPU based job
Date Tue, 10 Oct 2017 09:42:00 GMT

     [ https://issues.apache.org/jira/browse/MESOS-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ambuj Pandey updated MESOS-8065:
--------------------------------
    Description: 
I tried changing the "docker-no-executor-framework" to require GPU resource and use mesos
containerizer with a docker image of nvidia/cuda.

This worked for me when i ran this framework with the master IP of one installed with DC/OS.
But when I manually compiled Mesos (1.4/1.5) and started agent/master on another machine I
face this issue : "Implicitly declining offers in ACCEPT call as the launch operation specified
no tasks." 
I've tested the nvidia/cuda image to work on that target machine using nvidia-docker.

Configuration:
Master -
./bin/mesos-master.sh pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp  --filter_gpu_resources=false

Agent (on the same node) -
./bin/mesos-agent.sh --master=pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp --isolation="filesystem/linux,cgroups/devices,gpu/nvidia"

I1010 14:48:44.106006 20372 sched.cpp:336] New master detected at master@127.0.1.1:5050
I1010 14:48:44.107053 20372 sched.cpp:351] No credentials provided. Attempting to register
without authentication
I1010 14:48:44.113246 20365 sched.cpp:747] Framework registered with 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
Registered!
.Starting task 0 on pas-lab-server6
Hello
Task 0 is in state 3
. . . . . .

The log on master says:
{code:java}
I1010 14:48:44.420087 19698 master.cpp:7766] Sending 1 offers to framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
(Docker No Executor Framework (C++)) at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
W1010 14:48:44.422901 19701 master.cpp:3911] Implicitly declining offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
] in ACCEPT call for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 as the launch operation
specified no tasks
I1010 14:48:44.423539 19701 master.cpp:9389] Removing offer 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
I1010 14:48:44.423607 19701 master.cpp:4196] Processing ACCEPT call for offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
] on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 at slave(1)@127.0.1.1:5051 (pas-lab-server6)
for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 (Docker No Executor Framework (C++))
at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
I1010 14:48:44.425109 19691 hierarchical.cpp:887] Updated allocation of framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 from gpus(allocated: *):2; cpus(allocated:
*):40; mem(allocated: *):63421; disk(allocated: *):868675; ports(allocated: *):[31000-32000]
to gpus(allocated: *):2; cpus(allocated: *):40; mem(allocated: *):63421; disk(allocated: *):868675;
ports(allocated: *):[31000-32000]

{code}


  was:

I tried changing the "docker-no-executor-framework" to require GPU resource and use mesos
containerizer with a docker image of nvidia/cuda.

This worked for me when i ran this framework with the master IP of one installed with DC/OS.
But when I manually compiled Mesos (1.4/1.5) and started agent/master on another machine I
face this issue : "Implicitly declining offers in ACCEPT call as the launch operation specified
no tasks."

Configuration:
Master -
./bin/mesos-master.sh pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp  --filter_gpu_resources=false

Agent (on the same node) -
./bin/mesos-agent.sh --master=pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp --isolation="filesystem/linux,cgroups/devices,gpu/nvidia"

I1010 14:48:44.106006 20372 sched.cpp:336] New master detected at master@127.0.1.1:5050
I1010 14:48:44.107053 20372 sched.cpp:351] No credentials provided. Attempting to register
without authentication
I1010 14:48:44.113246 20365 sched.cpp:747] Framework registered with 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
Registered!
.Starting task 0 on pas-lab-server6
Hello
Task 0 is in state 3
. . . . . .

The log on master says:
{code:java}
I1010 14:48:44.420087 19698 master.cpp:7766] Sending 1 offers to framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
(Docker No Executor Framework (C++)) at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
W1010 14:48:44.422901 19701 master.cpp:3911] Implicitly declining offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
] in ACCEPT call for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 as the launch operation
specified no tasks
I1010 14:48:44.423539 19701 master.cpp:9389] Removing offer 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
I1010 14:48:44.423607 19701 master.cpp:4196] Processing ACCEPT call for offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
] on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 at slave(1)@127.0.1.1:5051 (pas-lab-server6)
for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 (Docker No Executor Framework (C++))
at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
I1010 14:48:44.425109 19691 hierarchical.cpp:887] Updated allocation of framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 from gpus(allocated: *):2; cpus(allocated:
*):40; mem(allocated: *):63421; disk(allocated: *):868675; ports(allocated: *):[31000-32000]
to gpus(allocated: *):2; cpus(allocated: *):40; mem(allocated: *):63421; disk(allocated: *):868675;
ports(allocated: *):[31000-32000]

{code}



> Implicitly declining offers in ACCEPT call as the launch operation specified no tasks
- for GPU based job
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: MESOS-8065
>                 URL: https://issues.apache.org/jira/browse/MESOS-8065
>             Project: Mesos
>          Issue Type: Bug
>          Components: docker, framework, gpu
>    Affects Versions: 1.4.0
>         Environment: Ubuntu 14.04
>            Reporter: Ambuj Pandey
>              Labels: docker, gpu, newbie
>
> I tried changing the "docker-no-executor-framework" to require GPU resource and use mesos
containerizer with a docker image of nvidia/cuda.
> This worked for me when i ran this framework with the master IP of one installed with
DC/OS. But when I manually compiled Mesos (1.4/1.5) and started agent/master on another machine
I face this issue : "Implicitly declining offers in ACCEPT call as the launch operation specified
no tasks." 
> I've tested the nvidia/cuda image to work on that target machine using nvidia-docker.
> Configuration:
> Master -
> ./bin/mesos-master.sh pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp  --filter_gpu_resources=false
> Agent (on the same node) -
> ./bin/mesos-agent.sh --master=pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp
--isolation="filesystem/linux,cgroups/devices,gpu/nvidia"
> I1010 14:48:44.106006 20372 sched.cpp:336] New master detected at master@127.0.1.1:5050
> I1010 14:48:44.107053 20372 sched.cpp:351] No credentials provided. Attempting to register
without authentication
> I1010 14:48:44.113246 20365 sched.cpp:747] Framework registered with 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
> Registered!
> .Starting task 0 on pas-lab-server6
> Hello
> Task 0 is in state 3
> . . . . . .
> The log on master says:
> {code:java}
> I1010 14:48:44.420087 19698 master.cpp:7766] Sending 1 offers to framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
(Docker No Executor Framework (C++)) at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
> W1010 14:48:44.422901 19701 master.cpp:3911] Implicitly declining offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
] in ACCEPT call for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 as the launch operation
specified no tasks
> I1010 14:48:44.423539 19701 master.cpp:9389] Removing offer 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
> I1010 14:48:44.423607 19701 master.cpp:4196] Processing ACCEPT call for offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
] on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 at slave(1)@127.0.1.1:5051 (pas-lab-server6)
for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 (Docker No Executor Framework (C++))
at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
> I1010 14:48:44.425109 19691 hierarchical.cpp:887] Updated allocation of framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 from gpus(allocated: *):2; cpus(allocated:
*):40; mem(allocated: *):63421; disk(allocated: *):868675; ports(allocated: *):[31000-32000]
to gpus(allocated: *):2; cpus(allocated: *):40; mem(allocated: *):63421; disk(allocated: *):868675;
ports(allocated: *):[31000-32000]
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message