Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B2B0A18CCE for ; Tue, 22 Dec 2015 11:42:20 +0000 (UTC) Received: (qmail 13642 invoked by uid 500); 22 Dec 2015 11:42:20 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 13577 invoked by uid 500); 22 Dec 2015 11:42:20 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 13566 invoked by uid 99); 22 Dec 2015 11:42:20 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2015 11:42:20 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 950521A02CF for ; Tue, 22 Dec 2015 11:42:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.549 X-Spam-Level: *** X-Spam-Status: No, score=3.549 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, NORMAL_HTTP_TO_IP=0.001, RP_MATCHES_RCVD=-0.554, SPF_PASS=-0.001, TRACKER_ID=1.102, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id tvITdRvKrNy5 for ; Tue, 22 Dec 2015 11:42:04 +0000 (UTC) Received: from smtp2.ba.infn.it (smtp2.ba.infn.it [192.135.10.91]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 9A65F20185 for ; Tue, 22 Dec 2015 11:42:03 +0000 (UTC) Received: from dhcp193.ba.infn.it (dhcp193.ba.infn.it [192.167.91.193]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp2.ba.infn.it (Postfix) with ESMTPSA id 74E8625DD2; Tue, 22 Dec 2015 12:41:54 +0100 (CET) Content-Type: multipart/signed; boundary="Apple-Mail=_30817854-0805-49E3-AAC4-F356473FBF78"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Mesos fetcher in dockerized slave From: Marica Antonacci In-Reply-To: <6A68B80A-1485-4836-9776-94163475D977@ba.infn.it> Date: Tue, 22 Dec 2015 12:41:46 +0100 Cc: tim@mesosphere.io Message-Id: References: <1147A296-CB9E-43F8-AB8C-CBF96D125C99@ba.infn.it> <8A1C0ED7-C9E3-4418-A1B2-0528B4FC34F8@ba.infn.it> <91874EDD-9CEC-41C6-A4D4-92301DF1D754@ba.infn.it> <6A68B80A-1485-4836-9776-94163475D977@ba.infn.it> To: user@mesos.apache.org X-Mailer: Apple Mail (2.1878.6) --Apple-Mail=_30817854-0805-49E3-AAC4-F356473FBF78 Content-Type: multipart/alternative; boundary="Apple-Mail=_06993C81-525D-4ACE-89A7-D1B16E4DFA17" --Apple-Mail=_06993C81-525D-4ACE-89A7-D1B16E4DFA17 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Dear all, I have not solved this issue yet. Please, can anyone run the same test = and let me know if the fetcher is correctly invoked?=20 The test is really simple, just try to start a dockerized app (see json = definition file below) through marathon on a mesos slave running in a = docker container started with the option =E2=80=94docker_mesos_image=3D. I would appreciate very much any feedback.=20 Sample Marathon app: {=20 "id": "test-app", "container": { "type": "DOCKER", "docker": { "image": "libmesos/ubuntu" } }, "cpus": 1, "mem": 512, "uris": [ = "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv" ], "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; = done"=20 } Docker run command to start dockerized mesos slave: # docker run -d MESOS_HOSTNAME=3D -e MESOS_IP=3D -e = MESOS_MASTER=3Dzk://:2181,:2181,:2181/mesos -e = MESOS_CONTAINERIZERS=3Ddocker,mesos -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave -v = /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host mesos-slave Thank you very much in advance! Best regards, Marica Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci = ha scritto: > Dear Tim, >=20 > I have collected some information from my test environment, starting = the slave container with and without the =E2=80=94docker_mesos_image = startup flag. Please let me know if you need further input. Thank you = very much for your support! >=20 > Using the flag =E2=80=94docker_mesos_image: >=20 > root@mesos-slave:~# docker ps > CONTAINER ID IMAGE COMMAND = CREATED STATUS PORTS NAMES > b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 = minutes ago Up 2 minutes = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d > da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 = minutes ago Up 2 minutes = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor > 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 = minutes ago Up 3 minutes slave >=20 > root@mesos-slave:~# docker logs slave > I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 > I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 > I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user = 'root' > I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; = mem(*):32 in work directory = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' > I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 > I1219 18:03:38.335907 19480 docker.cpp:762] Starting container = 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework = '246b272b-d649-47c0-88ca-6b1ff35f437a-0000' > I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to = '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks= /246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67= a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/fo= rked.pid' > I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for = executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from = executor(1)@90.147.170.246:39381 > I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating = container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed = to update is identical to existing resources > I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at = executor(1)@90.147.170.246:39381 > I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from = executor(1)@90.147.170.246:39381 > I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received = status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) = for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 > I1219 18:03:42.427276 19483 status_update_manager.cpp:826] = Checkpointing UPDATE for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 > I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050 > I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement = for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to = executor(1)@90.147.170.246:39381 > I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received = status update acknowledgement (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 > I1219 18:03:42.577263 19481 status_update_manager.cpp:826] = Checkpointing ACK for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >=20 > root@mesos-slave:~# docker logs = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor > = --container=3D"mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a= -4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" --stop_timeout=3D"0ns" > = --container=3D"mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a= -4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" --stop_timeout=3D"0ns" > I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0 > I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave = db70e09f-f39d-491c-8480-73d9858c140b-S0 > Registered docker executor on 90.147.170.246 > Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >=20 > root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor bash > root@mesos-slave:/# ls -R /tmp/ > hsperfdata_root/ mesos/ =20 > root@mesos-slave:/# ls -R /tmp/mesos/ > /tmp/mesos/: > slaves >=20 > /tmp/mesos/slaves: > db70e09f-f39d-491c-8480-73d9858c140b-S0 >=20 > /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0: > frameworks >=20 > /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks: > 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >=20 > = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000: > executors >=20 > = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors: > test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >=20 > = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5= -b1cf-fa163e920cd0: > runs >=20 > = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5= -b1cf-fa163e920cd0/runs: > d965f59b-cc1a-4081-95d2-f3370214c84d >=20 > = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5= -b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d: > stderr stdout > =20 >=20 > root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d bash > root@mesos-slave:/# env > HOSTNAME=3Dmesos-slave > HOST=3D90.147.170.246 > PORT0=3D31220 > PORT_10000=3D31220 > MESOS_TASK_ID=3Dtest-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 > PATH=3D/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > PWD=3D/ > PORTS=3D31220 > = MESOS_CONTAINER_NAME=3Dmesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f= 59b-cc1a-4081-95d2-f3370214c84d > SHLVL=3D1 > HOME=3D/ > MARATHON_APP_ID=3D/test-app > MARATHON_APP_DOCKER_IMAGE=3Dlibmesos/ubuntu > MARATHON_APP_VERSION=3D2015-12-19T18:03:37.542Z > PORT=3D31220 > MESOS_SANDBOX=3D/mnt/mesos/sandbox > _=3D/usr/bin/env > root@mesos-slave:/# ls -R $MESOS_SANDBOX > /mnt/mesos/sandbox: > stderr stdout >=20 >=20 > root@mesos-slave:/# ls /var/log/mesos* > /var/log/mesos-slave.INFO = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.206= 47 >=20 > /var/log/mesos: >=20 >=20 > Disabling the flag =E2=80=94docker_mesos_image the fetcher is called, = log is created, the file is downloaded in the sandbox: >=20 > root@mesos-slave:~# docker exec -it slave bash > root@mesos-slave:/#=20 > root@mesos-slave:/#=20 > root@mesos-slave:/#=20 > root@mesos-slave:/# ls /var/log/mesos* > /var/log/mesos-fetcher.INFO = /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.2= 0157 /var/log/mesos-slave.INFO = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.201= 24 >=20 > /var/log/mesos: > root@mesos-slave:/#=20 > root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO=20 > Log file created at: 2015/12/19 18:16:21 > Running on machine: mesos-slave > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg > I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging = started! > I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: = {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-7= 3d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"= value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practi= cals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e0= 9f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1= ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/= runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"} > I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' > I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into = the sandbox directory > I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' > I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv' > I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv' > root@mesos-slave:/#=20 >=20 > root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47= f58970b4 bash > root@mesos-slave:/#=20 > root@mesos-slave:/#=20 > root@mesos-slave:/# ls /mnt/mesos/sandbox/ =20 > snoqualmie.csv stderr stdout > root@mesos-slave:/#=20 >=20 >=20 > Best Regards, > Marica >=20 >=20 >=20 > Il giorno 19/dic/2015, alle ore 02:12, Tim Chen ha = scritto: >=20 >> HI Marica, >>=20 >> Did you see the fetcher invoked at all from the Slave logs? Doesn't = seem possible we don't pass down the URI flags and if the fetcher failed = the container launch should have failed too. >>=20 >> Also another possible situation is that web UI not really showing the = exact content of the sandbox, can you actually go into the directory and = see if the files are there? >>=20 >> Thanks, >>=20 >> Tim >>=20 >> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci = wrote: >> Hi Tim, >>=20 >> looking at the sandbox I can see only the stderr and stout file (see = the attached screenshot). If I remove =E2=80=94docker_mesos_image (and = therefore the executor is run inside the slave container) the file = specified in the URI field is shown in the sandbox. >> Did you verify that the fetcher is called when using the = =E2=80=94docker_mesos_image flag ? >>=20 >> Thanks a lot for your feedback >> Marica >>=20 >> >>=20 >> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen = ha scritto: >>=20 >>> Hi Marica, >>>=20 >>> It should work as we fetch all the files before we launch the = executor and place them in the sandbox, and we mount the sandbox into = that container as well. >>>=20 >>> How did you verify that the file is not downloaded?=20 >>>=20 >>> Tim >>>=20 >>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci = wrote: >>> Hi Grzegorz, >>>=20 >>> I=E2=80=99m using this command line for docker run >>>=20 >>> # docker run -d MESOS_HOSTNAME=3D -e MESOS_IP=3D = -e MESOS_MASTER=3Dzk://:2181,:2181,:2181/mesos = -e MESOS_CONTAINERIZERS=3Ddocker,mesos -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave -v = /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host mesos-slave >>>=20 >>> where mesos-slave is the image built from the docker file in this = repo https://github.com/maricaantonacci/mesos-slave-dev >>>=20 >>> I have tested successfully the deployment of dockerized applications = through Marathon and dockerized jobs through Chronos and also the = recovery seems to work fine with the flag docker_mesos_image. What is = not working for me is the fetcher: it seems that when the executor is = launched as separate container (thanks to the flag docker_mesos_image) = the information about the URIs to be downloaded is lost=E2=80=A6I hope = someone can help to understand if this a bug or I=E2=80=99 m missing = something. >>>=20 >>> Cheers, >>> Marica >>>=20 >>> =20 >>>=20 >>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk = ha scritto: >>>=20 >>>> I've tried to use this flag, but cannot really run any container = when this flag is set.=20 >>>> I've raised this issue here: = https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and = here: = https://github.com/mesosphere/docker-containers/issues/6#issuecomment-1553= 64351 but sadly no one was able to help me... >>>>=20 >>>> pt., 18.12.2015 o 11:33 u=C5=BCytkownik Marica Antonacci = napisa=C5=82: >>>> OK, the problem I spotted is related to the usage of the flag = =E2=80=94docker_mesos_image that allows the executor to=20 >>>>=20 >>>>=20 >>>> --docker_mesos_image=3DVALUE The docker image used to launch = this mesos slave instance. If an image is specified, the docker = containerizer assumes the slave is running in a docker container, and = launches executors with docker containers in order to recover them when = the slave restarts and recovers. >>>> Has anyone used this flag and tested the behavior of the fetcher? >>>>=20 >>>> Thank you >>>> Marica >>>>=20 >>>>=20 >>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao = ha scritto: >>>>=20 >>>>> no docker_mesos_image flag in my docker run, and the docker image = is build by myself. >>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci = : >>>>=20 >>>>> Yes, I did check inside the container and the csv file was not = downloaded as shown also by the app details (see the screenshot below). >>>>>=20 >>>>> Are you running your slave with the --docker_mesos_image flag? Can = you please provide me the docker run command you are using to run your = dockerized slave? =20 >>>>>=20 >>>>> Thank you very much >>>>=20 >>>>> Marica >>>>>=20 >>>>>=20 >>>>> >>>>=20 >>>>>=20 >>>>>=20 >>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao = ha scritto: >>>>>=20 >>>>>> Hi Marica, >>>>>>=20 >>>>>> use your test-app json, i can run it correctly, the csv is truely = download by mesos slave. please check mesos-master:5050 to check the = task detail download files. >>>>>>=20 >>>>>> you describe the app container why not found the csv, because the = csv is download in slave container's folder, not in app container. so if = you run=20 >>>>>>=20 >>>>>> cd $MESOS_SANDBOX; >>>>>>=20 >>>>>> the folder in app container is default value:=20 >>>>>> MESOS_SANDBOX=3D/mnt/mesos/sandbox >>>>>>=20 >>>>>> but in real world, the sandbox is in slave container, not in app = container. >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci = : >>>>>> Thank you very much, >>>>>>=20 >>>>>> I=E2=80=99m using a sample application definition file, just for = testing purpose: >>>>>>=20 >>>>>> {=20 >>>>>> "id": "test-app", >>>>>> "container": { >>>>>> "type": "DOCKER", >>>>>> "docker": { >>>>>> "image": "libmesos/ubuntu" >>>>>> } >>>>>> }, >>>>>> "cpus": 1, >>>>>> "mem": 512, >>>>>> "uris": [ = "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv" ], >>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u = +%T; done"=20 >>>>>> } >>>>>>=20 >>>>>> Here is the docker run command line: >>>>>>=20 >>>>>> # docker run -d -e MESOS_HOSTNAME=3D -e = MESOS_IP=3D -e = MESOS_MASTER=3Dzk://:2181,:2181,:2181/mesos -e = MESOS_CONTAINERIZERS=3Ddocker,mesos \ >>>>>> -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave=20 >>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock --name slave --net host = --privileged --pid host mesos-slave >>>>>>=20 >>>>>>=20 >>>>>> As already mentioned, if I remove the environment variable = MESOS_docker_mesos_image the fetcher works fine and I can see the file = snoqualmie.csv inside the sandbox. >>>>>>=20 >>>>>> Thank you again! I=E2=80=99m looking forward to hearing about = your outcomes. >>>>>> Best regards, >>>>>> Marica >>>>>>=20 >>>>>>=20 >>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao = ha scritto: >>>>>>=20 >>>>>>> Hi Marica, >>>>>>>=20 >>>>>>> Could you please give a sample marathon json, i can test it = asap. >>>>>>>=20 >>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci = : >>>>>>> No, using the socket: >>>>>>>=20 >>>>>>> -v /var/run/docker.sock:/var/run/docker.sock >>>>>>>=20 >>>>>>>=20 >>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao = ha scritto: >>>>>>>=20 >>>>>>>> docker in docker mode? >>>>>>>>=20 >>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci = : >>>>>>>> Dear all, >>>>>>>>=20 >>>>>>>> I'm testing the URIs fetching mechanism for both Marathon = applications and Chronos jobs and I have found that if the slave is = running inside a docker container (using docker_mesos_image startup = flag) and you submit the deployment of a dockerized application or job = the fetcher step is not performed. On the other hand, if I request the = deployment of a non-dockerized application, the URIs are correctly = fetched. Moreover, if I don=E2=80=99t provide the docker_mesos_image = flag, the fetcher works fine again for both dockerized and = non-dockerized applications. >>>>>>>>=20 >>>>>>>> Therefore, it seems that the information about the URIs gets = lost when the dockerized mesos slave spawns the executor docker = container that in turn launches the application docker container=E2=80=A6H= as anyone seen this problem before? I would like to know if there is a = workaround or a fixing. >>>>>>>>=20 >>>>>>>> Thanks a lot in advance for you help >>>>>>>> Best Regards, >>>>>>>> Marica >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> ---------------------------------------------------------- >>>>>>>> Marica ANTONACCI >>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>> Via Orabona 4 >>>>>>>> 70126 Bari - ITALY >>>>>>>> Phone +39 080 5443244 >>>>>>>> Skype: marica.antonacci >>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> --=20 >>>>>>>> Deshi Xiao >>>>>>>> Twitter: xds2000 >>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>=20 >>>>>>> ---------------------------------------------------------- >>>>>>> Marica ANTONACCI >>>>>>> INFN - National Institute of Nuclear Physics >>>>>>> Via Orabona 4 >>>>>>> 70126 Bari - ITALY >>>>>>> Phone +39 080 5443244 >>>>>>> Skype: marica.antonacci >>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> --=20 >>>>>>> Deshi Xiao >>>>>>> Twitter: xds2000 >>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>=20 >>>>>> ---------------------------------------------------------- >>>>>> Marica ANTONACCI >>>>>> INFN - National Institute of Nuclear Physics >>>>>> Via Orabona 4 >>>>>> 70126 Bari - ITALY >>>>>> Phone +39 080 5443244 >>>>>> Skype: marica.antonacci >>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> --=20 >>>>>> Deshi Xiao >>>>>> Twitter: xds2000 >>>>>> E-mail: xiaods(AT)gmail.com >>>>>> <=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.16.png><=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 = 2015-12-18 =E4=B8=8B=E5=8D=884.55.30.png><=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85= =A7 2015-12-18 =E4=B8=8B=E5=8D=884.55.38.png> >>>>>=20 >>>>> ---------------------------------------------------------- >>>>> Marica ANTONACCI >>>>> INFN - National Institute of Nuclear Physics >>>>> Via Orabona 4 >>>>> 70126 Bari - ITALY >>>>> Phone +39 080 5443244 >>>>> Skype: marica.antonacci >>>>> e-mail marica.antonacci@ba.infn.it >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> --=20 >>>>> Deshi Xiao >>>>> Twitter: xds2000 >>>>> E-mail: xiaods(AT)gmail.com >>>>=20 >>>>=20 >>>> ---------------------------------------------------------- >>>> Marica ANTONACCI >>>> INFN - National Institute of Nuclear Physics >>>> Via Orabona 4 >>>> 70126 Bari - ITALY >>>> Phone +39 080 5443244 >>>> Skype: marica.antonacci >>>> e-mail marica.antonacci@ba.infn.it >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>=20 >>> ---------------------------------------------------------- >>> Marica ANTONACCI >>> INFN - National Institute of Nuclear Physics >>> Via Orabona 4 >>> 70126 Bari - ITALY >>> Phone +39 080 5443244 >>> Skype: marica.antonacci >>> e-mail marica.antonacci@ba.infn.it >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>=20 >> ---------------------------------------------------------- >> Marica ANTONACCI >> INFN - National Institute of Nuclear Physics >> Via Orabona 4 >> 70126 Bari - ITALY >> Phone +39 080 5443244 >> Skype: marica.antonacci >> e-mail marica.antonacci@ba.infn.it >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >=20 > ---------------------------------------------------------- > Marica ANTONACCI > INFN - National Institute of Nuclear Physics > Via Orabona 4 > 70126 Bari - ITALY > Phone +39 080 5443244 > Skype: marica.antonacci > e-mail marica.antonacci@ba.infn.it >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 ---------------------------------------------------------- Marica ANTONACCI INFN - National Institute of Nuclear Physics Via Orabona 4 70126 Bari - ITALY Phone +39 080 5443244 Skype: marica.antonacci e-mail marica.antonacci@ba.infn.it --Apple-Mail=_06993C81-525D-4ACE-89A7-D1B16E4DFA17 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Dear = all,

I have not solved this issue yet. Please, can = anyone run the same test and let me know if the fetcher is correctly = invoked? 
The test is really simple, just try to start a = dockerized app (see json definition file below) through marathon on a = mesos slave running in a docker container started with the option = =E2=80=94docker_mesos_image=3D<mesos slave image>.
I = would appreciate very much any = feedback. 

Sample Marathon = app:
{ 
 "id": "test-app",
 "container": = {
   "type": "DOCKER",
   "docker": = {
     "image": "libmesos/ubuntu"
  =  }
 },
 "cpus": 1,
 "mem": = 512,
 "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pract= icals/snoqualmie.csv" ],
 "cmd": "cd $MESOS_SANDBOX; ls = -latr; while sleep 10; do date -u +%T; = done" 
}

Docker run command to start = dockerized mesos slave:

# docker run -d = MESOS_HOSTNAME=3D<SLAVE_IP> -e MESOS_IP=3D<SLAVE_IP> -e = MESOS_MASTER=3Dzk://<node-1>:2181,<node-2>:2181,<node-3>= :2181/mesos -e MESOS_CONTAINERIZERS=3Ddocker,mesos = -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e = MESOS_LOG_DIR=3D/var/log -e MESOS_docker_mesos_image=3Dmesos-slave -v = /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host = mesos-slave

Thank you very much in = advance!
Best = regards,
Marica

Il giorno 19/dic/2015, = alle ore 19:32, Marica Antonacci <marica.antonacci@ba.infn.it> ha scritto:

Dear = Tim,

I have collected some information from my = test environment, starting the slave container with and without the = =E2=80=94docker_mesos_image startup flag. Please let me know if you need = further input. Thank you very much for your = support!

Using the flag = =E2=80=94docker_mesos_image:

root@mesos-slave:~# = docker ps
CONTAINER ID        IMAGE =               COMMAND    =               CREATED     =         STATUS          =     PORTS               = NAMES
b30cea22a07c        libmesos/ubuntu   =   "/bin/sh -c 'cd $MESO"   2 minutes ago       = Up 2 minutes                =             = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d
da9c78ec5727        mesos-slave   =       "/bin/sh -c '/usr/lib"   2 minutes ago   =     Up 2 minutes            =                 = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor
150f78fbf327        mesos-slave =         "/entrypoint.sh /usr/"   3 minutes ago =       Up 3 minutes          =                   = slave

root@mesos-slave:~# docker logs = slave
I1219 18:03:38.308544 19476 = slave.cpp:1294] Got assigned task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user = 'root'
I1219 18:03:38.327221 19476 = slave.cpp:4999] Launching executor = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; = mem(*):32 in work directory = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
I1219 18:03:38.330817 19476 slave.cpp:1628] = Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for = executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.335907 19480 docker.cpp:762] Starting container = 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework = '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid = 19590 to = '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks= /246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67= a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/fo= rked.pid'
I1219 18:03:40.189172 19479 = slave.cpp:2405] Got registration for executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from = executor(1)@90.147.170.246:39381
I1219 = 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container = 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update = is identical to existing resources
I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at = executor(1)@90.147.170.246:39381
I1219 = 18:03:42.425606 19483 slave.cpp:2762] Handling status update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from = executor(1)@90.147.170.246:39381
I1219 = 18:03:42.426415 19483 status_update_manager.cpp:322] Received status = update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for = task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.427276 19483 status_update_manager.cpp:826] = Checkpointing UPDATE for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
=
I1219 18:03:42.521852 19483 slave.cpp:3011] = Sending acknowledgement for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to = executor(1)@90.147.170.246:39381
I1219 = 18:03:42.576373 19481 status_update_manager.cpp:394] Received status = update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for = task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.577263 19481 status_update_manager.cpp:826] = Checkpointing ACK for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000

root@mesos-slave:~# docker logs = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor
--container=3D"mesos-db70e09f-f39d-491c-8480-73d9= 858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" = --stop_timeout=3D"0ns"
--container=3D"mesos-db70e09f-f39d-491c-8480-73d= 9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" --stop_timeout=3D"0ns"
I1219 18:03:40.177598   =   6 exec.cpp:136] Version: 0.26.0
I1219 = 18:03:40.192060    10 exec.cpp:210] Executor registered = on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
Registered docker = executor on 90.147.170.246
Starting task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0

root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor bash
root@mesos-slave:/# ls -R /tmp/
hsperfdata_root/ mesos/          =  
root@mesos-slave:/# ls -R /tmp/mesos/
/tmp/mesos/:
slaves

/tmp/mesos/slaves:
db70e09f-f39d-491c-8480-73d9858c140b-S0

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
frameworks

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks:
246b272b-d649-47c0-88ca-6b1ff35f437a-0000

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
executors

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a= 67a-11e5-b1cf-fa163e920cd0:
runs

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a= 67a-11e5-b1cf-fa163e920cd0/runs:
d965f59b-cc1a-4081-95d2-f3370214c84d

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a= 67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
stderr  stdout
 

root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d bash
root@mesos-slave:/# env
HOSTNAME=3Dmesos-slave
HOST=3D90.147.170.246
PORT0=3D31220
PORT_10000=3D31220
MESOS_TASK_ID=3Dtest-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
PATH=3D/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bi= n
PWD=3D/
PORTS=3D31220
MESOS_CONTAINER_NAME=3Dmesos-db70e09f-f39d-491c-8480-73d9858c140b-= S0.d965f59b-cc1a-4081-95d2-f3370214c84d
SHLVL=3D1
HOME=3D/
MARATHON_APP_ID=3D/test-app
MARATHON_APP_DOCKER_IMAGE=3Dlibmesos/ubuntu
MARATHON_APP_VERSION=3D2015-12-19T18:03:37.542Z
PORT=3D31220
MESOS_SANDBOX=3D/mnt/mesos/sandbox
_=3D/usr/bin/env
root@mesos-slave:/# ls -R = $MESOS_SANDBOX
/mnt/mesos/sandbox:
stderr = stdout


root@mesos-slave:/# ls /var/log/mesos*
/var/log/mesos-slave.INFO  = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.206= 47

/var/log/mesos:


Disa= bling the flag =E2=80=94docker_mesos_image the fetcher is called, log is = created, the file is downloaded in the = sandbox:

root@mesos-slave:~# docker exec -it slave = bash
root@mesos-slave:/# 
root@mesos-slave:/# 
root@mesos-slave:/# 
root@mesos-slave:/# ls = /var/log/mesos*
/var/log/mesos-fetcher.INFO  = /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.2= 0157  /var/log/mesos-slave.INFO  = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.201= 24

/var/log/mesos:
root@mesos-slave:/# 
root@mesos-slave:/# cat = /var/log/mesos-fetcher.INFO 
Log file created at: 2015/12/19 = 18:16:21
Running on machine: mesos-slave
Log line format: [IWEF]mmdd = hh:mm:ss.uuuuuu threadid file:line] msg
I1219 18:16:21.127075 20157 = logging.cpp:172] INFO level logging started!
I1219 18:16:21.127499 20157 = fetcher.cpp:422] Fetcher Info: = {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-7= 3d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"= value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/le= ctures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/= mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b27= 2b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
I1219 = 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-prac= ticals/snoqualmie.csv'
I1219 18:16:21.129576 20157 fetcher.cpp:248] = Fetching directly into the sandbox directory
I1219 18:16:21.129607 20157 = fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-prac= ticals/snoqualmie.csv'
I1219 18:16:21.129636 20157 fetcher.cpp:132] = Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-prac= ticals/snoqualmie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv'
I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-prac= ticals/snoqualmie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv'
root@mesos-slave:/# 

root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47= f58970b4 bash
root@mesos-slave:/# 
root@mesos-slave:/# 
root@mesos-slave:/# ls = /mnt/mesos/sandbox/  
snoqualmie.csv stderr = stdout
root@mesos-slave:/# 


Best = Regards,
Marica



Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <tim@mesosphere.io> ha = scritto:

HI Marica,

Did you see = the fetcher invoked at all from the Slave logs? Doesn't seem possible we = don't pass down the URI flags and if the fetcher failed the container = launch should have failed too.

Also another = possible situation is that web UI not really showing the exact content = of the sandbox, can you actually go into the directory and see if the = files are = there?

Thanks,

Tim
=

On Fri, = Dec 18, 2015 at 4:23 PM, Marica Antonacci <marica.antonacci@ba.infn.it> = wrote:
Hi Tim,

looking at the = sandbox I can see only the stderr and stout file (see the attached = screenshot). If I remove =E2=80=94docker_mesos_image (and therefore the = executor is run inside the slave container) the file specified in the = URI field is shown in the sandbox.
Did you verify that the = fetcher is called when using the =E2=80=94docker_mesos_image flag = ?

Thanks a lot for your = feedback
Marica

<Schermata = 2015-12-18 alle 10.15.58.png>

Il giorno 19/dic/2015, alle ore 00:25, Tim = Chen <tim@mesosphere.io> ha = scritto:

Hi = Marica,

It should work as we fetch all the files = before we launch the executor and place them in the sandbox, and we = mount the sandbox into that container as = well.

How did you verify that the file is not = downloaded? 

Tim

On Fri, Dec 18, = 2015 at 5:26 AM, Marica Antonacci <marica.antonacci@ba.infn.it> = wrote:
Hi Grzegorz,

I=E2=80= =99m using this command line for docker run

# = docker run -d MESOS_HOSTNAME=3D<SLAVE_IP> -e = MESOS_IP=3D<SLAVE_IP> -e = MESOS_MASTER=3Dzk://<node-1>:2181,<node-2>:2181,<node-3>= :2181/mesos -e MESOS_CONTAINERIZERS=3Ddocker,mesos -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave = -v /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host = mesos-slave

where mesos-slave is the image = built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev

I have tested successfully the deployment of = dockerized applications through Marathon and dockerized jobs through = Chronos and also the recovery seems to work fine with the flag = docker_mesos_image. What is not working for me is the fetcher: it seems = that when the executor is launched as separate container (thanks to the = flag docker_mesos_image) the information about the URIs to be downloaded = is lost=E2=80=A6I hope someone can help to understand if this a bug or = I=E2=80=99 m missing = something.

Cheers,
Marica

  

Il giorno = 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gregory90@gmail.com> ha = scritto:

I've tried = to use this flag, but cannot really run any container when this flag is = set. 

pt., = 18.12.2015 o 11:33 u=C5=BCytkownik Marica Antonacci <marica.antonacci@ba.infn.it> = napisa=C5=82:
OK, the problem I spotted is related = to the usage of the flag =E2=80=94docker_mesos_image that allows = the executor to 


--docker_mesos_image=3DVALUEThe docker image used to launch this mesos = slave instance. If an image is specified, the docker containerizer = assumes the slave is running in a docker container, and launches = executors with docker containers in order to recover them when the slave = restarts and recovers.
Has anyone used = this flag and tested the behavior of the = fetcher?

Thank = you
Marica


Il giorno = 18/dic/2015, alle ore 10:38, tommy xiao <xiaods@gmail.com> ha = scritto:

no docker_mesos_image flag in my docker run, =  and the docker image is build by myself.



2015-12-18 17:20 = GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
Yes, = I did check inside the container and the csv file was not downloaded as = shown also by the app details (see the screenshot = below).

Are you running your slave with the = --docker_mesos_image flag? Can you please provide me the docker run = command you are using to run your dockerized slave? =  

Thank you very = much
Marica


<Schermata 2015-12-18 alle = 10.15.58.png>


Il = giorno 18/dic/2015, alle ore 10:00, tommy xiao <xiaods@gmail.com> ha = scritto:

Hi= Marica,

use your test-app json, i can run it = correctly, the csv is truely download by mesos slave. please check = mesos-master:5050 to check the task detail download = files.

you describe the app container why not = found the csv, because the csv is download in slave container's folder, = not in app container. so if you run 

cd $MESOS_SANDBOX;

the folder in app container is default = value: 

MESOS_SANDBOX=3D/mnt/mesos/sandbox<= /span>

but in real world, = the sandbox is in slave container, not in app = container.



2015-12-18 16:11 = GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
Thank you very = much,

I=E2=80=99m using a sample application = definition file, just for testing = purpose:

{ 
 "id": = "test-app",
 "container": {
   "type": = "DOCKER",
   "docker": {
     "image": = "libmesos/ubuntu"
   }
 },
 "cpus": = 1,
 "mem": 512,
 "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pr= acticals/snoqualmie.csv" ],
 "cmd": "cd $MESOS_SANDBOX; = ls -latr; while sleep 10; do date -u +%T; = done" 
}

Here is the docker run command = line:

# docker run -d -e = MESOS_HOSTNAME=3D<SLAVE_IP> -e MESOS_IP=3D<SLAVE_IP> -e = MESOS_MASTER=3Dzk://<node-1>:2181,<node-2>:2181,<node-3>= :2181/mesos -e MESOS_CONTAINERIZERS=3Ddocker,mesos = \
                =       -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e = MESOS_LOG_DIR=3D/var/log -e = MESOS_docker_mesos_image=3Dmesos-slave 
    =                   = -v /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock --name slave --net host = --privileged --pid host = mesos-slave


As already = mentioned, if I remove the environment variable MESOS_docker_mesos_image = the fetcher works fine and I can see the file snoqualmie.csv inside the = sandbox.

Thank you again! I=E2=80=99m looking = forward to hearing about your outcomes.
Best = regards,
Marica


Il = giorno 18/dic/2015, alle ore 04:51, tommy xiao <xiaods@gmail.com> ha = scritto:

Hi = Marica,

Could you please give a sample marathon json, = i can test it asap.

2015-12-18 5:11 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
No, using = the socket:

-v = /var/run/docker.sock:/var/run/docker.sock

Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xiaods@gmail.com> ha = scritto:

docker in = docker mode?

2015-12-17 19:08 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
Dear = all,

I'm testing the URIs fetching mechanism for both = Marathon applications and Chronos jobs and I have found that if the = slave is running inside a docker container = (using docker_mesos_image startup flag) and you submit the = deployment of a dockerized application or job the fetcher step is not = performed. On the other hand, if I request the deployment of a = non-dockerized application, the URIs are correctly fetched. Moreover, if = I don=E2=80=99t provide the docker_mesos_image flag, the fetcher works = fine again for both dockerized and non-dockerized = applications.

Therefore, it seems that the = information about the URIs gets lost when the dockerized mesos slave = spawns the executor docker container that in turn launches the = application docker container=E2=80=A6Has anyone seen this problem = before? I would like to know if there is a workaround or a = fixing.

Thanks a lot in advance for you = help
Best = Regards,
Marica


----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







--=
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







--
Deshi Xiao
Twitter: = xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







--=
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
<=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.16.png><=E5=B1=8F=E5=B9=95=E5=BF= =AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.30.png><=E5=B1=8F=E5=B9=95=E5=BF= =AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.38.png>

= 
----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







-- =
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 





----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 






----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it











----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN =
- National Institute of Nuclear Physics
Via =
Orabona 4
70126 Bari - ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it
= --Apple-Mail=_06993C81-525D-4ACE-89A7-D1B16E4DFA17-- --Apple-Mail=_30817854-0805-49E3-AAC4-F356473FBF78 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEXDCCBFgw ggNAoAMCAQICAwCERjANBgkqhkiG9w0BAQsFADAuMQswCQYDVQQGEwJJVDENMAsGA1UEChMESU5G TjEQMA4GA1UEAxMHSU5GTiBDQTAeFw0xNTA4MjcxMjQwMTFaFw0xNjA4MjYxMjQwMTFaMGUxCzAJ BgNVBAYTAklUMQ0wCwYDVQQKEwRJTkZOMR0wGwYDVQQLExRQZXJzb25hbCBDZXJ0aWZpY2F0ZTEN MAsGA1UEBxMEQmFyaTEZMBcGA1UEAxMQTWFyaWNhIEFudG9uYWNjaTCCASIwDQYJKoZIhvcNAQEB BQADggEPADCCAQoCggEBAOhg+3XzXvCUsWnkCvdDahsjrQ2OEZjfxtreK5ceeFfJEPWBr/b3Bhzo A6KqCu6Ay2Uhp1k3pgzdiuCIHiDjAznrpMutrCpThX7OXZwI8xp5B2UUaK+sSS5XASY54+ClThY7 Ix2IDlLj+FCw7ZG28iHRMDtdc6Hb3bxpVM3C1HjCL8qdCyeIrgTF+ghWhl7fedKW8rE3tqHnjjn0 mBqb7Jw1soYK3e61qgnQc7aB3osM5olf9yDuAQ/arhsLqS76XzLY0Hvy4iH2W728XyAe1zMv2szk zbgjyxEhoW0rY8tvsewpn4NCoRB2nowEwDJKJTvHwNXqnH/JrnPp8IQ9OvcCAwEAAaOCAUYwggFC MAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEF BQcDBDA9BgNVHR8ENjA0MDKgMKAuhixodHRwOi8vc2VjdXJpdHkuZmkuaW5mbi5pdC9DQS9JTkZO Q0FfY3JsLmRlcjAlBgNVHSAEHjAcMAwGCisGAQQB0SMKAQcwDAYKKoZIhvdMBQICATAdBgNVHQ4E FgQUBdtWP9C9wUcP6X3i7+/V65V6+/IwVgYDVR0jBE8wTYAU0WLzs3dyyC778nkabzdOJ58T1SCh MqQwMC4xCzAJBgNVBAYTAklUMQ0wCwYDVQQKEwRJTkZOMRAwDgYDVQQDEwdJTkZOIENBggEAMCYG A1UdEQQfMB2BG21hcmljYS5hbnRvbmFjY2lAYmEuaW5mbi5pdDANBgkqhkiG9w0BAQsFAAOCAQEA gWA1/J088VDH4Sn2YBUC9lu4mmY35F+CAL90rmWOGRR1FbNv7qlJyNvJBsGIcUhZ2GDYioKLzoPy 7RTTaoIkX6Fen22mCSIyhAvVPxCxY4sbDlxU3y9mXZCllZhkrPT553GHsZohQhaSRQJ9hne5Mpse 2c4QhYz/FftrqnQWdIjUhqmXRBdmnGU8GYZX9MUys/H6v8pOmd/cmX3EZJnGF9raKx6/mooHCXPQ 98z24EZj3uDqhojhDrRvzvq+FivuOyJRQ0Opw7bHTvbNgylB9ritf41eVPmiJBrkwObn1N3jkxFc LhN0vhzspXw0HalDES0Sye/2ALbayZ05CasgPTGCAkowggJGAgEBMDUwLjELMAkGA1UEBhMCSVQx DTALBgNVBAoTBElORk4xEDAOBgNVBAMTB0lORk4gQ0ECAwCERjAJBgUrDgMCGgUAoIHrMBgGCSqG SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE1MTIyMjExNDE0NlowIwYJKoZI hvcNAQkEMRYEFJGZRw9vuIauarwZ9HUrZgfCgbEyMEQGCSsGAQQBgjcQBDE3MDUwLjELMAkGA1UE BhMCSVQxDTALBgNVBAoTBElORk4xEDAOBgNVBAMTB0lORk4gQ0ECAwCERjBGBgsqhkiG9w0BCRAC CzE3oDUwLjELMAkGA1UEBhMCSVQxDTALBgNVBAoTBElORk4xEDAOBgNVBAMTB0lORk4gQ0ECAwCE RjANBgkqhkiG9w0BAQEFAASCAQBov6pHB7pA4vMHBKkf94jZQA7EMuZLq44zJXIbeM7A8bQrKqDK 5fDyAdzw+vrgViAlSwa0ZsKg3jYj/gw/7Mrw/ojVxBXB1lAsBAc90av0/Ua1FiQ+nB4Q8dzKeyHf /kbt5Ur+EqtlBqlQ/Wwz/XbAyBD/pXiUURpxtfsR5FXpJLZOfCSapZjtU3uSVrRip7Fc8icQJsX8 OiIRtM2oh+RAtYVyGGqbmbqBmlIYa7pid+98Zef1HHNqeb9rfOWRM6QLjyXSN8oXQBUhwhAipRhw BVqwfiE/ji74nGYyzO25ofMXHtoSVbmTRdf+oxJvUX6JHUY2BgRq64JXP12PNg+lAAAAAAAA --Apple-Mail=_30817854-0805-49E3-AAC4-F356473FBF78--