Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3D65A200BCB for ; Thu, 24 Nov 2016 13:53:01 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3BB3E160B1E; Thu, 24 Nov 2016 12:53:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5FA19160B1F for ; Thu, 24 Nov 2016 13:53:00 +0100 (CET) Received: (qmail 42651 invoked by uid 500); 24 Nov 2016 12:52:59 -0000 Mailing-List: contact issues-help@aurora.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@aurora.apache.org Delivered-To: mailing list issues@aurora.apache.org Received: (qmail 42515 invoked by uid 99); 24 Nov 2016 12:52:59 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Nov 2016 12:52:59 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 646872C03DE for ; Thu, 24 Nov 2016 12:52:59 +0000 (UTC) Date: Thu, 24 Nov 2016 12:52:59 +0000 (UTC) From: "Kostiantyn Bokhan (JIRA)" To: issues@aurora.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (AURORA-1830) Unknown exception initializing sandbox MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 24 Nov 2016 12:53:01 -0000 Kostiantyn Bokhan created AURORA-1830: ----------------------------------------- Summary: Unknown exception initializing sandbox Key: AURORA-1830 URL: https://issues.apache.org/jira/browse/AURORA-1830 Project: Aurora Issue Type: Bug Components: Executor Affects Versions: 0.16.0 Reporter: Kostiantyn Bokhan When launching a job using the Mesos containerizer and a docker image, the = sandbox setup fails with the following error: {quote} FAILED =E2=80=A2 Unknown exception initializing sandbox: [Errno 2] No such= file or directory {quote} Aurora file: {quote} # run the script python =3D Process( name =3D 'python', cmdline =3D 'python --version') # describe the task python_task =3D Task( processes =3D [python], resources =3D Resources(cpu =3D 1, ram =3D 1*GB, disk=3D8*GB)) jobs =3D [ Service(cluster =3D 'MY Cluster', environment =3D 'devel', role =3D 'root', name =3D 'python', task =3D python_task, container =3D Mesos( image =3D DockerImage (name =3D 'python', ta= g =3D '2'))) ] {quote} __main__.log: {quote} Log file created at: 2016/11/24 14:45:44 Running on machine: gnode1 [DIWEF]mmdd hh:mm:ss.uuuuuu pid file:line] msg Command line: /var/lib/mesos/slave/slaves/195fbdc8-6720-443b-b036-7fa5608b2= 7cc-S24/frameworks/195fbdc8-6720-443b-b036-7fa5608b27cc-0014/executors/ther= mos-root-devel-python-0-e33ad106-90dd-481a-8d45-c320990b67d8/runs/e25e2e98-= 0b65-4e9f-a86d-13a18dff01bc/thermos_executor --announcer-ensemble 127.0.0.1= :2181 I1124 14:45:44.041621 25610 executor_base.py:45] Executor [None]: registere= d() called with: I1124 14:45:44.042294 25610 executor_base.py:45] Executor [None]: Execut= orInfo: executor_id { value: "thermos-root-devel-python-0-e33ad106-90dd-481a-8d45-c320990b67d8" } resources { name: "cpus" type: SCALAR scalar { value: 0.25 } role: "*" } resources { name: "mem" type: SCALAR scalar { value: 128.0 } role: "*" } command { uris { value: "/usr/bin/thermos_executor" executable: true } value: "${MESOS_SANDBOX=3D.}/thermos_executor --announcer-ensemble 127.0.= 0.1:2181" } framework_id { value: "195fbdc8-6720-443b-b036-7fa5608b27cc-0014" } name: "AuroraExecutor" source: "root.devel.python.0" container { type: MESOS volumes { container_path: "taskfs" mode: RO image { type: DOCKER docker { name: python:2" } } } mesos { } } labels { labels { key: "source" value: "root.devel.python.0" } } I1124 14:45:44.042458 25610 executor_base.py:45] Executor [None]: Framew= orkInfo: user: "root" name: "Aurora" id { value: "195fbdc8-6720-443b-b036-7fa5608b27cc-0014" } failover_timeout: 1814400.0 checkpoint: true hostname: "vnode7" capabilities { type: GPU_RESOURCES } I1124 14:45:44.043046 25610 executor_base.py:45] Executor [None]: SlaveI= nfo: hostname: "000.000.00.001" resources { name: "gpus" type: SCALAR scalar { value: 2.0 } role: "*" } resources { name: "ports" type: RANGES ranges { range { begin: 1025 end: 2180 } range { begin: 2182 end: 3887 } range { begin: 3889 end: 5049 } range { begin: 5052 end: 8079 } range { begin: 8082 end: 8180 } range { begin: 8182 end: 32000 } } role: "*" } resources { name: "disk" type: SCALAR scalar { value: 428201.0 } role: "*" } resources { name: "cpus" type: SCALAR scalar { value: 8.0 } role: "*" } resources { name: "mem" type: SCALAR scalar { value: 14957.0 } role: "*" } attributes { name: "hostname" type: TEXT text { value: "gnode1" } } attributes { name: "ip" type: TEXT text { value: "000.000.00.001" } } attributes { name: "rack" type: TEXT text { value: "gpu" } } attributes { name: "gputype" type: TEXT text { value: "titanz" } } id { value: "195fbdc8-6720-443b-b036-7fa5608b27cc-S24" } checkpoint: true port: 5051 I1124 14:45:44.043673 25610 executor_base.py:45] Executor [None]: launchTas= k got task: root/devel/python:root-devel-python-0-e33ad106-90dd-481a-8d45-c= 320990b67d8 I1124 14:45:44.044601 25610 executor_base.py:45] Executor [195fbdc8-6720-44= 3b-b036-7fa5608b27cc-S24]: Updating root-devel-python-0-e33ad106-90dd-481a-= 8d45-c320990b67d8 =3D> STARTING I1124 14:45:44.044718 25610 executor_base.py:45] Executor [195fbdc8-6720-44= 3b-b036-7fa5608b27cc-S24]: Reason: Initializing sandbox. F1124 14:45:44.049196 25610 aurora_executor.py:85] Unknown exception initia= lizing sandbox: [Errno 2] No such file or directory I1124 14:45:44.049439 25610 executor_base.py:45] Executor [195fbdc8-6720-44= 3b-b036-7fa5608b27cc-S24]: Updating root-devel-python-0-e33ad106-90dd-481a-= 8d45-c320990b67d8 =3D> FAILED I1124 14:45:44.049519 25610 executor_base.py:45] Executor [195fbdc8-6720-44= 3b-b036-7fa5608b27cc-S24]: Reason: Unknown exception initializing sandbo= x: [Errno 2] No such file or directory I1124 14:45:49.152787 25610 thermos_executor_main.py:299] MesosExecutorDriv= er.run() has finished. {quote} stderr {quote} I1124 14:45:43.559283 25614 fetcher.cpp:498] Fetcher Info: {"cache_director= y":"\/tmp\/mesos\/fetch\/slaves\/195fbdc8-6720-443b-b036-7fa5608b27cc-S24\/= root","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":= true,"value":"\/usr\/bin\/thermos_executor"}}],"sandbox_directory":"\/var\/= lib\/mesos\/slave\/slaves\/195fbdc8-6720-443b-b036-7fa5608b27cc-S24\/framew= orks\/195fbdc8-6720-443b-b036-7fa5608b27cc-0014\/executors\/thermos-root-de= vel-python-0-e33ad106-90dd-481a-8d45-c320990b67d8\/runs\/e25e2e98-0b65-4e9f= -a86d-13a18dff01bc","user":"root"} I1124 14:45:43.561226 25614 fetcher.cpp:409] Fetching URI '/usr/bin/thermos= _executor' I1124 14:45:43.561242 25614 fetcher.cpp:250] Fetching directly into the san= dbox directory I1124 14:45:43.561266 25614 fetcher.cpp:187] Fetching URI '/usr/bin/thermos= _executor' I1124 14:45:43.561285 25614 fetcher.cpp:167] Copying resource with command:= cp '/usr/bin/thermos_executor' '/var/lib/mesos/slave/slaves/195fbdc8-6720-4= 43b-b036-7fa5608b27cc-S24/frameworks/195fbdc8-6720-443b-b036-7fa5608b27cc-0= 014/executors/thermos-root-devel-python-0-e33ad106-90dd-481a-8d45-c320990b6= 7d8/runs/e25e2e98-0b65-4e9f-a86d-13a18dff01bc/thermos_executor' I1124 14:45:43.569787 25614 fetcher.cpp:547] Fetched '/usr/bin/thermos_exec= utor' to '/var/lib/mesos/slave/slaves/195fbdc8-6720-443b-b036-7fa5608b27cc-= S24/frameworks/195fbdc8-6720-443b-b036-7fa5608b27cc-0014/executors/thermos-= root-devel-python-0-e33ad106-90dd-481a-8d45-c320990b67d8/runs/e25e2e98-0b65= -4e9f-a86d-13a18dff01bc/thermos_executor' twitter.common.app debug: Initializing: twitter.common.log (Logging subsyst= em.) Writing log files to disk in /var/lib/mesos/slave/slaves/195fbdc8-6720-443b= -b036-7fa5608b27cc-S24/frameworks/195fbdc8-6720-443b-b036-7fa5608b27cc-0014= /executors/thermos-root-devel-python-0-e33ad106-90dd-481a-8d45-c320990b67d8= /runs/e25e2e98-0b65-4e9f-a86d-13a18dff01bc I1124 14:45:44.033974 25610 exec.cpp:161] Version: 1.0.0 I1124 14:45:44.040127 25639 exec.cpp:236] Executor registered on agent 195f= bdc8-6720-443b-b036-7fa5608b27cc-S24 FATAL] Unknown exception initializing sandbox: [Errno 2] No such file or di= rectory twitter.common.app debug: Shutting application down. twitter.common.app debug: Running exit function for twitter.common.log (Log= ging subsystem.) twitter.common.app debug: Finishing up module teardown. twitter.common.app debug: Active thread: <_MainThread(MainThread, started= 139772146038592)> twitter.common.app debug: Active thread (daemon): <_DummyThread(Dummy-2, = started daemon 139771946940160)> twitter.common.app debug: Exiting cleanly. {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)