Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 90F1318EFF for ; Wed, 2 Mar 2016 08:56:53 +0000 (UTC) Received: (qmail 14789 invoked by uid 500); 2 Mar 2016 08:56:52 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 14723 invoked by uid 500); 2 Mar 2016 08:56:52 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 14713 invoked by uid 99); 2 Mar 2016 08:56:52 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2016 08:56:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 14E18C3B76 for ; Wed, 2 Mar 2016 08:56:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.802 X-Spam-Level: ** X-Spam-Status: No, score=2.802 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, TRACKER_ID=1.102, WEIRD_PORT=0.001] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id MNxKdXe9LKq4 for ; Wed, 2 Mar 2016 08:56:46 +0000 (UTC) Received: from smtp2.ba.infn.it (smtp2.ba.infn.it [192.135.10.91]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 833365FAFF for ; Wed, 2 Mar 2016 08:56:45 +0000 (UTC) Received: from dhcp193.ba.infn.it (dhcp193.ba.infn.it [192.167.91.193]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp2.ba.infn.it (Postfix) with ESMTPSA id 15CBA20066; Wed, 2 Mar 2016 09:56:38 +0100 (CET) Content-Type: multipart/alternative; boundary="Apple-Mail=_A91B2C61-71E0-4E2E-9C67-EAA72A60A480" Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Mesos fetcher in dockerized slave From: Marica Antonacci In-Reply-To: Date: Wed, 2 Mar 2016 09:56:36 +0100 Cc: Shuai Lin , Timothy Chen Message-Id: References: <1147A296-CB9E-43F8-AB8C-CBF96D125C99@ba.infn.it> <8A1C0ED7-C9E3-4418-A1B2-0528B4FC34F8@ba.infn.it> <91874EDD-9CEC-41C6-A4D4-92301DF1D754@ba.infn.it> <6A68B80A-1485-4836-9776-94163475D977@ba.infn.it> <537DA9F1-F257-419F-999A-47629F007C27@ba.infn.it> <3C823D0B-DF75-4854-82CF-A4B2DC1A742D@mesosphere.io> <2E3F3D91-B448-40F5-A78B-075363C0C9A9@ba.infn.it> To: user@mesos.apache.org X-Mailer: Apple Mail (2.1878.6) --Apple-Mail=_A91B2C61-71E0-4E2E-9C67-EAA72A60A480 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Thank you, I did not receive the notification (don=E2=80=99t know = why=E2=80=A6maybe I missed it) Thanks to all! Best, Marica Il giorno 02/mar/2016, alle ore 09:38, haosdent ha = scritto: > I saw MESOS-4249 have already fixed in 0.28? >=20 > On Wed, Mar 2, 2016 at 4:31 PM, Marica Antonacci = wrote: > Dear Shuai and Tim, >=20 > any news about the patch for MESOS-4743? > Can I ask you when it will be officially released? >=20 > Thank you in advance > Best regards, > Marica >=20 > Il giorno 24/feb/2016, alle ore 11:23, Shuai Lin = ha scritto: >=20 >> ping @Tim, I think this bug also affects = https://issues.apache.org/jira/browse/MESOS-4743 . >>=20 >>=20 >>=20 >> On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin = wrote: >> The testing of this case requires to build a docker image for = mesos-slave, so it seems not practical to add a test case for it in the = mesos tests.=20 >>=20 >> Anyway, here is the scripts I use for testing this issue: = https://gist.github.com/lins05/14455e92f37e91fd46ff >>=20 >> On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin = wrote: >> Hi Tim, >>=20 >> The review is here: https://reviews.apache.org/r/42390/ , would you = please take a look? >>=20 >> Regards, >> Shuai >>=20 >> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin = wrote: >> Hi Maria and Tim,=20 >>=20 >> I'm setting up a test case for this scenario that would fail, after = which I'll begin on fixing it.=20 >>=20 >> Is it feasible to include the fixing in the new release? >>=20 >> I'm not sure, does 0.27 have an estimated release date now? >>=20 >>=20 >> Regards, >> Shuai >>=20 >> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen = wrote: >> I can shepherd no problem. >>=20 >> Tim >>=20 >> On Dec 25, 2015, at 4:32 PM, Shuai Lin = wrote: >>=20 >>> I'll work on it. @Tim could you shepherd it? >>>=20 >>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci = wrote: >>> Hi Tim and Shuai, >>>=20 >>> thank you very much for your reply. I have opened a JIRA issue on = this: https://issues.apache.org/jira/browse/MESOS-4249 >>> I hope it will be patched soon :)=20 >>>=20 >>> Best regards, >>> Marica >>>=20 >>>=20 >>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen = ha scritto: >>>=20 >>>> Hi Marica/Shuai, >>>>=20 >>>> Sorry haven't been able to spend the time to repro, but looks like = Shuai confirmed it. >>>>=20 >>>> Can one of you file a JIRA? >>>>=20 >>>> Thanks! >>>>=20 >>>> Tim >>>>=20 >>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin = wrote: >>>> Hi Marica, >>>>=20 >>>> I can reproduce the problem exactly as you described in the first = email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment = variable set, the fetcher works just fine; With it, the fetcher steps = seems skipped. This looks like a bug to me. >>>>=20 >>>> Regards, >>>> Shuai >>>>=20 >>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci = wrote: >>>> Dear all, >>>>=20 >>>> I have not solved this issue yet. Please, can anyone run the same = test and let me know if the fetcher is correctly invoked?=20 >>>> The test is really simple, just try to start a dockerized app (see = json definition file below) through marathon on a mesos slave running in = a docker container started with the option =E2=80=94docker_mesos_image=3D<= mesos slave image>. >>>> I would appreciate very much any feedback.=20 >>>>=20 >>>> Sample Marathon app: >>>> {=20 >>>> "id": "test-app", >>>> "container": { >>>> "type": "DOCKER", >>>> "docker": { >>>> "image": "libmesos/ubuntu" >>>> } >>>> }, >>>> "cpus": 1, >>>> "mem": 512, >>>> "uris": [ = "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv" ], >>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u = +%T; done"=20 >>>> } >>>>=20 >>>> Docker run command to start dockerized mesos slave: >>>>=20 >>>> # docker run -d MESOS_HOSTNAME=3D -e MESOS_IP=3D = -e MESOS_MASTER=3Dzk://:2181,:2181,:2181/mesos = -e MESOS_CONTAINERIZERS=3Ddocker,mesos -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave -v = /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host mesos-slave >>>>=20 >>>> Thank you very much in advance! >>>> Best regards, >>>> Marica >>>>=20 >>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci = ha scritto: >>>>=20 >>>>> Dear Tim, >>>>>=20 >>>>> I have collected some information from my test environment, = starting the slave container with and without the =E2=80=94docker_mesos_im= age startup flag. Please let me know if you need further input. Thank = you very much for your support! >>>>>=20 >>>>> Using the flag =E2=80=94docker_mesos_image: >>>>>=20 >>>>> root@mesos-slave:~# docker ps >>>>> CONTAINER ID IMAGE COMMAND = CREATED STATUS PORTS NAMES >>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 = minutes ago Up 2 minutes = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d >>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 = minutes ago Up 2 minutes = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor >>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 = minutes ago Up 3 minutes slave >>>>>=20 >>>>> root@mesos-slave:~# docker logs slave >>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user = 'root' >>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; = mem(*):32 in work directory = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container = 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework = '246b272b-d649-47c0-88ca-6b1ff35f437a-0000' >>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid = 19590 to = '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks= /246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67= a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/fo= rked.pid' >>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for = executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from = executor(1)@90.147.170.246:39381 >>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating = container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed = to update is identical to existing resources >>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at = executor(1)@90.147.170.246:39381 >>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from = executor(1)@90.147.170.246:39381 >>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] = Received status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] = Checkpointing UPDATE for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050 >>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending = acknowledgement for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to = executor(1)@90.147.170.246:39381 >>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] = Received status update acknowledgement (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] = Checkpointing ACK for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>=20 >>>>> root@mesos-slave:~# docker logs = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor >>>>> = --container=3D"mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a= -4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" --stop_timeout=3D"0ns" >>>>> = --container=3D"mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a= -4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" --stop_timeout=3D"0ns" >>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0 >>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on = slave db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>>> Registered docker executor on 90.147.170.246 >>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>=20 >>>>> root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor bash >>>>> root@mesos-slave:/# ls -R /tmp/ >>>>> hsperfdata_root/ mesos/ =20 >>>>> root@mesos-slave:/# ls -R /tmp/mesos/ >>>>> /tmp/mesos/: >>>>> slaves >>>>>=20 >>>>> /tmp/mesos/slaves: >>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>>>=20 >>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0: >>>>> frameworks >>>>>=20 >>>>> = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks: >>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>=20 >>>>> = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000: >>>>> executors >>>>>=20 >>>>> = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors: >>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>=20 >>>>> = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5= -b1cf-fa163e920cd0: >>>>> runs >>>>>=20 >>>>> = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5= -b1cf-fa163e920cd0/runs: >>>>> d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>=20 >>>>> = /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b2= 72b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5= -b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d: >>>>> stderr stdout >>>>> =20 >>>>>=20 >>>>> root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d bash >>>>> root@mesos-slave:/# env >>>>> HOSTNAME=3Dmesos-slave >>>>> HOST=3D90.147.170.246 >>>>> PORT0=3D31220 >>>>> PORT_10000=3D31220 >>>>> MESOS_TASK_ID=3Dtest-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>> PATH=3D/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>> PWD=3D/ >>>>> PORTS=3D31220 >>>>> = MESOS_CONTAINER_NAME=3Dmesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f= 59b-cc1a-4081-95d2-f3370214c84d >>>>> SHLVL=3D1 >>>>> HOME=3D/ >>>>> MARATHON_APP_ID=3D/test-app >>>>> MARATHON_APP_DOCKER_IMAGE=3Dlibmesos/ubuntu >>>>> MARATHON_APP_VERSION=3D2015-12-19T18:03:37.542Z >>>>> PORT=3D31220 >>>>> MESOS_SANDBOX=3D/mnt/mesos/sandbox >>>>> _=3D/usr/bin/env >>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX >>>>> /mnt/mesos/sandbox: >>>>> stderr stdout >>>>>=20 >>>>>=20 >>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>> /var/log/mesos-slave.INFO = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.206= 47 >>>>>=20 >>>>> /var/log/mesos: >>>>>=20 >>>>>=20 >>>>> Disabling the flag =E2=80=94docker_mesos_image the fetcher is = called, log is created, the file is downloaded in the sandbox: >>>>>=20 >>>>> root@mesos-slave:~# docker exec -it slave bash >>>>> root@mesos-slave:/#=20 >>>>> root@mesos-slave:/#=20 >>>>> root@mesos-slave:/#=20 >>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>> /var/log/mesos-fetcher.INFO = /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.2= 0157 /var/log/mesos-slave.INFO = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.201= 24 >>>>>=20 >>>>> /var/log/mesos: >>>>> root@mesos-slave:/#=20 >>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO=20 >>>>> Log file created at: 2015/12/19 18:16:21 >>>>> Running on machine: mesos-slave >>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] = msg >>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging = started! >>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: = {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-7= 3d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"= value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practi= cals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e0= 9f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1= ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/= runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"} >>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' >>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly = into the sandbox directory >>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' >>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource = from = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv' >>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched = 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv' >>>>> root@mesos-slave:/#=20 >>>>>=20 >>>>> root@mesos-slave:~# docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47= f58970b4 bash >>>>> root@mesos-slave:/#=20 >>>>> root@mesos-slave:/#=20 >>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/ =20 >>>>> snoqualmie.csv stderr stdout >>>>> root@mesos-slave:/#=20 >>>>>=20 >>>>>=20 >>>>> Best Regards, >>>>> Marica >>>>>=20 >>>>>=20 >>>>>=20 >>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen = ha scritto: >>>>>=20 >>>>>> HI Marica, >>>>>>=20 >>>>>> Did you see the fetcher invoked at all from the Slave logs? = Doesn't seem possible we don't pass down the URI flags and if the = fetcher failed the container launch should have failed too. >>>>>>=20 >>>>>> Also another possible situation is that web UI not really showing = the exact content of the sandbox, can you actually go into the directory = and see if the files are there? >>>>>>=20 >>>>>> Thanks, >>>>>>=20 >>>>>> Tim >>>>>>=20 >>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci = wrote: >>>>>> Hi Tim, >>>>>>=20 >>>>>> looking at the sandbox I can see only the stderr and stout file = (see the attached screenshot). If I remove =E2=80=94docker_mesos_image = (and therefore the executor is run inside the slave container) the file = specified in the URI field is shown in the sandbox. >>>>>> Did you verify that the fetcher is called when using the = =E2=80=94docker_mesos_image flag ? >>>>>>=20 >>>>>> Thanks a lot for your feedback >>>>>> Marica >>>>>>=20 >>>>>> >>>>>>=20 >>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen = ha scritto: >>>>>>=20 >>>>>>> Hi Marica, >>>>>>>=20 >>>>>>> It should work as we fetch all the files before we launch the = executor and place them in the sandbox, and we mount the sandbox into = that container as well. >>>>>>>=20 >>>>>>> How did you verify that the file is not downloaded?=20 >>>>>>>=20 >>>>>>> Tim >>>>>>>=20 >>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci = wrote: >>>>>>> Hi Grzegorz, >>>>>>>=20 >>>>>>> I=E2=80=99m using this command line for docker run >>>>>>>=20 >>>>>>> # docker run -d MESOS_HOSTNAME=3D -e = MESOS_IP=3D -e = MESOS_MASTER=3Dzk://:2181,:2181,:2181/mesos -e = MESOS_CONTAINERIZERS=3Ddocker,mesos -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave -v = /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host mesos-slave >>>>>>>=20 >>>>>>> where mesos-slave is the image built from the docker file in = this repo https://github.com/maricaantonacci/mesos-slave-dev >>>>>>>=20 >>>>>>> I have tested successfully the deployment of dockerized = applications through Marathon and dockerized jobs through Chronos and = also the recovery seems to work fine with the flag docker_mesos_image. = What is not working for me is the fetcher: it seems that when the = executor is launched as separate container (thanks to the flag = docker_mesos_image) the information about the URIs to be downloaded is = lost=E2=80=A6I hope someone can help to understand if this a bug or I=E2=80= =99 m missing something. >>>>>>>=20 >>>>>>> Cheers, >>>>>>> Marica >>>>>>>=20 >>>>>>> =20 >>>>>>>=20 >>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk = ha scritto: >>>>>>>=20 >>>>>>>> I've tried to use this flag, but cannot really run any = container when this flag is set.=20 >>>>>>>> I've raised this issue here: = https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and = here: = https://github.com/mesosphere/docker-containers/issues/6#issuecomment-1553= 64351 but sadly no one was able to help me... >>>>>>>>=20 >>>>>>>> pt., 18.12.2015 o 11:33 u=C5=BCytkownik Marica Antonacci = napisa=C5=82: >>>>>>>> OK, the problem I spotted is related to the usage of the flag = =E2=80=94docker_mesos_image that allows the executor to=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> --docker_mesos_image=3DVALUE The docker image used to launch = this mesos slave instance. If an image is specified, the docker = containerizer assumes the slave is running in a docker container, and = launches executors with docker containers in order to recover them when = the slave restarts and recovers. >>>>>>>> Has anyone used this flag and tested the behavior of the = fetcher? >>>>>>>>=20 >>>>>>>> Thank you >>>>>>>> Marica >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao = ha scritto: >>>>>>>>=20 >>>>>>>>> no docker_mesos_image flag in my docker run, and the docker = image is build by myself. >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>=20 >>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci = : >>>>>>>>=20 >>>>>>>>> Yes, I did check inside the container and the csv file was not = downloaded as shown also by the app details (see the screenshot below). >>>>>>>>>=20 >>>>>>>>> Are you running your slave with the --docker_mesos_image flag? = Can you please provide me the docker run command you are using to run = your dockerized slave? =20 >>>>>>>>>=20 >>>>>>>>> Thank you very much >>>>>>>>=20 >>>>>>>>> Marica >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> >>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao = ha scritto: >>>>>>>>>=20 >>>>>>>>>> Hi Marica, >>>>>>>>>>=20 >>>>>>>>>> use your test-app json, i can run it correctly, the csv is = truely download by mesos slave. please check mesos-master:5050 to check = the task detail download files. >>>>>>>>>>=20 >>>>>>>>>> you describe the app container why not found the csv, because = the csv is download in slave container's folder, not in app container. = so if you run=20 >>>>>>>>>>=20 >>>>>>>>>> cd $MESOS_SANDBOX; >>>>>>>>>>=20 >>>>>>>>>> the folder in app container is default value:=20 >>>>>>>>>> MESOS_SANDBOX=3D/mnt/mesos/sandbox >>>>>>>>>>=20 >>>>>>>>>> but in real world, the sandbox is in slave container, not in = app container. >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci = : >>>>>>>>>> Thank you very much, >>>>>>>>>>=20 >>>>>>>>>> I=E2=80=99m using a sample application definition file, just = for testing purpose: >>>>>>>>>>=20 >>>>>>>>>> {=20 >>>>>>>>>> "id": "test-app", >>>>>>>>>> "container": { >>>>>>>>>> "type": "DOCKER", >>>>>>>>>> "docker": { >>>>>>>>>> "image": "libmesos/ubuntu" >>>>>>>>>> } >>>>>>>>>> }, >>>>>>>>>> "cpus": 1, >>>>>>>>>> "mem": 512, >>>>>>>>>> "uris": [ = "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualm= ie.csv" ], >>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date = -u +%T; done"=20 >>>>>>>>>> } >>>>>>>>>>=20 >>>>>>>>>> Here is the docker run command line: >>>>>>>>>>=20 >>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=3D -e = MESOS_IP=3D -e = MESOS_MASTER=3Dzk://:2181,:2181,:2181/mesos -e = MESOS_CONTAINERIZERS=3Ddocker,mesos \ >>>>>>>>>> -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave=20 >>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock --name slave --net host = --privileged --pid host mesos-slave >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> As already mentioned, if I remove the environment variable = MESOS_docker_mesos_image the fetcher works fine and I can see the file = snoqualmie.csv inside the sandbox. >>>>>>>>>>=20 >>>>>>>>>> Thank you again! I=E2=80=99m looking forward to hearing about = your outcomes. >>>>>>>>>> Best regards, >>>>>>>>>> Marica >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao = ha scritto: >>>>>>>>>>=20 >>>>>>>>>>> Hi Marica, >>>>>>>>>>>=20 >>>>>>>>>>> Could you please give a sample marathon json, i can test it = asap. >>>>>>>>>>>=20 >>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci = : >>>>>>>>>>> No, using the socket: >>>>>>>>>>>=20 >>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao = ha scritto: >>>>>>>>>>>=20 >>>>>>>>>>>> docker in docker mode? >>>>>>>>>>>>=20 >>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci = : >>>>>>>>>>>> Dear all, >>>>>>>>>>>>=20 >>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon = applications and Chronos jobs and I have found that if the slave is = running inside a docker container (using docker_mesos_image startup = flag) and you submit the deployment of a dockerized application or job = the fetcher step is not performed. On the other hand, if I request the = deployment of a non-dockerized application, the URIs are correctly = fetched. Moreover, if I don=E2=80=99t provide the docker_mesos_image = flag, the fetcher works fine again for both dockerized and = non-dockerized applications. >>>>>>>>>>>>=20 >>>>>>>>>>>> Therefore, it seems that the information about the URIs = gets lost when the dockerized mesos slave spawns the executor docker = container that in turn launches the application docker container=E2=80=A6H= as anyone seen this problem before? I would like to know if there is a = workaround or a fixing. >>>>>>>>>>>>=20 >>>>>>>>>>>> Thanks a lot in advance for you help >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> Marica >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>>=20 >>>>>>>>>>>> --=20 >>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>=20 >>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>> Via Orabona 4 >>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>>=20 >>>>>>>>>>> --=20 >>>>>>>>>>> Deshi Xiao >>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>=20 >>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>> Marica ANTONACCI >>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>> Via Orabona 4 >>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>> Skype: marica.antonacci >>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>>=20 >>>>>>>>>> --=20 >>>>>>>>>> Deshi Xiao >>>>>>>>>> Twitter: xds2000 >>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>> <=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.16.png><=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 = 2015-12-18 =E4=B8=8B=E5=8D=884.55.30.png><=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85= =A7 2015-12-18 =E4=B8=8B=E5=8D=884.55.38.png> >>>>>>>>>=20 >>>>>>>>> ---------------------------------------------------------- >>>>>>>>> Marica ANTONACCI >>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>> Via Orabona 4 >>>>>>>>> 70126 Bari - ITALY >>>>>>>>> Phone +39 080 5443244 >>>>>>>>> Skype: marica.antonacci >>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>>=20 >>>>>>>>> --=20 >>>>>>>>> Deshi Xiao >>>>>>>>> Twitter: xds2000 >>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> ---------------------------------------------------------- >>>>>>>> Marica ANTONACCI >>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>> Via Orabona 4 >>>>>>>> 70126 Bari - ITALY >>>>>>>> Phone +39 080 5443244 >>>>>>>> Skype: marica.antonacci >>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>=20 >>>>>>> ---------------------------------------------------------- >>>>>>> Marica ANTONACCI >>>>>>> INFN - National Institute of Nuclear Physics >>>>>>> Via Orabona 4 >>>>>>> 70126 Bari - ITALY >>>>>>> Phone +39 080 5443244 >>>>>>> Skype: marica.antonacci >>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>> ---------------------------------------------------------- >>>>>> Marica ANTONACCI >>>>>> INFN - National Institute of Nuclear Physics >>>>>> Via Orabona 4 >>>>>> 70126 Bari - ITALY >>>>>> Phone +39 080 5443244 >>>>>> Skype: marica.antonacci >>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>> ---------------------------------------------------------- >>>>> Marica ANTONACCI >>>>> INFN - National Institute of Nuclear Physics >>>>> Via Orabona 4 >>>>> 70126 Bari - ITALY >>>>> Phone +39 080 5443244 >>>>> Skype: marica.antonacci >>>>> e-mail marica.antonacci@ba.infn.it >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>>> ---------------------------------------------------------- >>>> Marica ANTONACCI >>>> INFN - National Institute of Nuclear Physics >>>> Via Orabona 4 >>>> 70126 Bari - ITALY >>>> Phone +39 080 5443244 >>>> Skype: marica.antonacci >>>> e-mail marica.antonacci@ba.infn.it >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>=20 >>> ---------------------------------------------------------- >>> Marica ANTONACCI >>> INFN - National Institute of Nuclear Physics >>> Via Orabona 4 >>> 70126 Bari - ITALY >>> Phone +39 080 5443244 >>> Skype: marica.antonacci >>> e-mail marica.antonacci@ba.infn.it >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>=20 >>=20 >>=20 >>=20 >=20 > ---------------------------------------------------------- > Marica ANTONACCI > INFN - National Institute of Nuclear Physics > Via Orabona 4 > 70126 Bari - ITALY > Phone +39 080 5443244 > Skype: marica.antonacci > e-mail marica.antonacci@ba.infn.it >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > --=20 > Best Regards, > Haosdent Huang ---------------------------------------------------------- Marica ANTONACCI INFN - National Institute of Nuclear Physics Via Orabona 4 70126 Bari - ITALY Phone +39 080 5443244 Skype: marica.antonacci e-mail marica.antonacci@ba.infn.it --Apple-Mail=_A91B2C61-71E0-4E2E-9C67-EAA72A60A480 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Thank = you, I did not receive the notification (don=E2=80=99t know why=E2=80=A6ma= ybe I missed it)

Thanks to = all!
Best,
Marica

Il giorno = 02/mar/2016, alle ore 09:38, haosdent <haosdent@gmail.com> ha = scritto:

I saw MESOS-4249 have already fixed = in 0.28?

On= Wed, Mar 2, 2016 at 4:31 PM, Marica Antonacci <marica.antonacci@ba.infn.it> = wrote:
Dear Shuai and = Tim,

any news about the patch for = MESOS-4743?
Can I ask you when it will be officially = released?

Thank you in advance
Best = regards,
Marica

Il = giorno 24/feb/2016, alle ore 11:23, Shuai Lin <linshuai2012@gmail.com> ha = scritto:

ping @Tim, = I think this bug also affects https://issues.apache.org/jira/browse/MESOS-4743 = .



On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <linshuai2012@gmail.com> = wrote:
The testing of this case = requires to build a docker image for mesos-slave, so it seems not = practical to add a test case for it in the mesos = tests. 

Anyway, here is the scripts I use for = testing this issue: https://gist.github.com/lins05/14455e92f37e91fd46ff<= /div>

On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <linshuai2012@gmail.com> = wrote:
Hi = Tim,

The review is here: https://reviews.apache.org/r/42390/ , would you = please take a = look?

Regards,
Shuai

On Sat, Jan 9, = 2016 at 9:42 AM, Shuai Lin <linshuai2012@gmail.com> = wrote:
Hi Maria and = Tim, 

I'm setting up a test case for this = scenario that would fail, after which I'll begin on fixing = it. 

Is it feasible to = include the fixing in the new = release?

I'm not sure, does 0.27 = have an estimated release date = now?


Regards,
Shua= i

On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <tim@mesosphere.io> wrote:
I can shepherd no = problem.

Tim

On Dec 25, 2015, at 4:32 PM, Shuai Lin <linshuai2012@gmail.com> = wrote:

I'll work = on it. @Tim could you shepherd it?

On Sat, Dec 26, = 2015 at 2:49 AM, Marica Antonacci <marica.antonacci@ba.infn.it> = wrote:
Hi = Tim and Shuai,

thank you very much for your reply. I = have opened a JIRA issue on this: https://issues.apache.org/jira/browse/MESOS-4249
=
I hope it will be patched soon = :) 

Best = regards,
Marica


Il = giorno 24/dic/2015, alle ore 17:54, Tim Chen <tim@mesosphere.io> ha = scritto:

Hi = Marica/Shuai,

Sorry haven't been able to spend the = time to repro, but looks like Shuai confirmed = it.

Can one of you file a = JIRA?

Thanks!

Tim
<= /div>

On Thu, = Dec 24, 2015 at 6:16 AM, Shuai Lin <linshuai2012@gmail.com> = wrote:
Hi = Marica,

I can reproduce the problem exactly as you = described in the first email of this thread. Without = `MESOS_DOCKER_MESOS_IMAGE` environment variable set, the fetcher works = just fine; With it, the fetcher steps seems skipped. This looks like a = bug to = me.

Regards,
Shuai

On Tue, Dec 22, = 2015 at 7:41 PM, Marica Antonacci <marica.antonacci@ba.infn.it> = wrote:
Dear = all,

I have not solved this issue yet. Please, can = anyone run the same test and let me know if the fetcher is correctly = invoked? 
The test is really simple, just try to start a = dockerized app (see json definition file below) through marathon on a = mesos slave running in a docker container started with the option = =E2=80=94docker_mesos_image=3D<mesos slave image>.
I = would appreciate very much any = feedback. 

Sample Marathon = app:
{ 
 "id": = "test-app",
 "container": {
   "type": = "DOCKER",
   "docker": {
     "image": = "libmesos/ubuntu"
   }
 },
 "cpus": = 1,
 "mem": 512,
 "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pr= acticals/snoqualmie.csv" ],
 "cmd": "cd $MESOS_SANDBOX; ls = -latr; while sleep 10; do date -u +%T; = done" 
}

Docker run command to = start dockerized mesos slave:

# docker = run -d MESOS_HOSTNAME=3D<SLAVE_IP> -e MESOS_IP=3D<SLAVE_IP> = -e = MESOS_MASTER=3Dzk://<node-1>:2181,<node-2>:2181,<node-3>= :2181/mesos -e MESOS_CONTAINERIZERS=3Ddocker,mesos = -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e = MESOS_LOG_DIR=3D/var/log -e MESOS_docker_mesos_image=3Dmesos-slave -v = /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host = mesos-slave

Thank you very much in = advance!
Best = regards,
Marica

Il giorno 19/dic/2015, = alle ore 19:32, Marica Antonacci <marica.antonacci@ba.infn.it> ha = scritto:

Dear Tim,

I = have collected some information from my test environment, starting the = slave container with and without the =E2=80=94docker_mesos_image startup = flag. Please let me know if you need further input. Thank you very much = for your support!

Using the flag = =E2=80=94docker_mesos_image:

root@mesos-slave:~# docker = ps
CONTAINER ID  =       IMAGE             =   COMMAND                =   CREATED             STATUS  =             PORTS       =         NAMES
b30cea22a07c  =       libmesos/ubuntu     "/bin/sh -c 'cd = $MESO"   2 minutes ago       Up 2 minutes  =                     =       = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d
da9c78ec5727  =       mesos-slave         "/bin/sh -c = '/usr/lib"   2 minutes ago       Up 2 minutes  =                     =       = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor
150f78fbf327  =       mesos-slave         = "/entrypoint.sh /usr/"   3 minutes ago       Up 3 = minutes                  =           slave

root@mesos-slave:= ~# docker logs slave
I1219 18:03:38.308544 19476 slave.cpp:1294] Got = assigned task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for = framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.314268 19476 slave.cpp:1410] = Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for = framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.316261 19476 paths.cpp:436] Trying = to chown = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user = 'root'
I1219 18:03:38.327221 19476 = slave.cpp:4999] Launching executor = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; = mem(*):32 in work directory = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing = task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.335907 19480 docker.cpp:762] = Starting container 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework = '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
I1219 18:03:40.084027 19482 docker.cpp:386] = Checkpointing pid 19590 to = '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks= /246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67= a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/fo= rked.pid'
I1219 18:03:40.189172 19479 = slave.cpp:2405] Got registration for executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
I1219 18:03:40.191141 19479 docker.cpp:1012] = Ignoring updating container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with = resources passed to update is identical to existing resources
I1219 18:03:40.192512 19479 slave.cpp:1793] Sending = queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor = 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381
I1219 18:03:42.425606 19483 slave.cpp:2762] = Handling status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
I1219 18:03:42.426415 19483 = status_update_manager.cpp:322] Received status update TASK_RUNNING = (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.427276 19483 = status_update_manager.cpp:826] Checkpointing UPDATE for status update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.521395 19483 slave.cpp:3087] = Forwarding the update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
I1219 18:03:42.521852 19483 slave.cpp:3011] Sending = acknowledgement for status update TASK_RUNNING (UUID: = 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
I1219 18:03:42.576373 19481 = status_update_manager.cpp:394] Received status update acknowledgement = (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.577263 19481 = status_update_manager.cpp:826] Checkpointing ACK for status update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework = 246b272b-d649-47c0-88ca-6b1ff35f437a-0000

root@mesos-slave:~# docker logs = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor
--container=3D"mesos-db70e09f-f39d-491c-8480-73d9= 858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" = --stop_timeout=3D"0ns"
--container=3D"mesos-db70e09f-f39d-491c-8480-73d= 9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" --docker=3D"docker" = --docker_socket=3D"/var/run/docker.sock" --help=3D"false" = --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" = --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" = --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c1= 40b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33= 70214c84d" --stop_timeout=3D"0ns"
I1219 18:03:40.177598   =   6 exec.cpp:136] Version: 0.26.0
I1219 = 18:03:40.192060    10 exec.cpp:210] Executor registered = on slave db70e09f-f39d-491c-8480-73d9858c140b-S0
Registered docker = executor on 90.147.170.246
Starting task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0

root@mesos-slave:~# = docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d.executor bash
root@mesos-slave:/# = ls -R /tmp/
hsperfdata_root/ = mesos/           
root@mesos-slave:/# = ls -R /tmp/mesos/
/tmp/mesos/:
slaves

=
/tmp/mesos/slaves:
db70e09f-f39d-491c-8480-73d9858c140b-S0

=
/tmp/mesos/slaves/db= 70e09f-f39d-491c-8480-73d9858c140b-S0:
frameworks

=
/tmp/mesos/slaves/db= 70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
246b272b-d649-47c0-88ca-6b1ff35f437a-0000

=
/tmp/mesos/slaves/db= 70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6= b1ff35f437a-0000:
executors

=
/tmp/mesos/slaves/db= 70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6= b1ff35f437a-0000/executors:
test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0

=
/tmp/mesos/slaves/db= 70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6= b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:<= /div>
runs

=
/tmp/mesos/slaves/db= 70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6= b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/r= uns:
d965f59b-cc1a-4081-95d2-f3370214c84d

=
/tmp/mesos/slaves/db= 70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6= b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/r= uns/d965f59b-cc1a-4081-95d2-f3370214c84d:
stderr  = stdout
 
root@mesos-slave:~# = docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f337= 0214c84d bash
root@mesos-slave:/# = env
HOSTNAME=3Dmesos-sla= ve
HOST=3D90.147.170.24= 6
PORT0=3D31220
<= div = style=3D"margin:0px;font-size:10px;font-family:Menlo">PORT_10000=3D31220
MESOS_TASK_ID=3Dtest= -app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
PATH=3D/usr/local/sb= in:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=3D/
PORTS=3D31220
<= div = style=3D"margin:0px;font-size:10px;font-family:Menlo">MESOS_CONTAINER_NAME= =3Dmesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f= 3370214c84d
SHLVL=3D1
HOME=3D/
MARATHON_APP_ID=3D/t= est-app
MARATHON_APP_DOCKER_= IMAGE=3Dlibmesos/ubuntu
MARATHON_APP_VERSION= =3D2015-12-19T18:03:37.542Z
PORT=3D31220
MESOS_SANDBOX=3D/mnt= /mesos/sandbox
_=3D/usr/bin/env
root@mesos-slave:/# = ls -R $MESOS_SANDBOX
/mnt/mesos/sandbox:<= /div>
stderr = stdout


root@mesos-slave:/# = ls /var/log/mesos*
/var/log/mesos-slave.INFO&nb= sp; = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.206= 47

=
/var/log/mesos:


Disabling the flag = =E2=80=94docker_mesos_image the fetcher is called, log is created, the = file is downloaded in the sandbox:

root@mesos-slave:~# = docker exec -it slave bash
root@mesos-slave:/#&= nbsp;
root@mesos-slave:/#&= nbsp;
root@mesos-slave:/#&= nbsp;
root@mesos-slave:/# = ls /var/log/mesos*
/var/log/mesos-fetcher.INFO&= nbsp; = /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.2= 0157  /var/log/mesos-slave.INFO&nb= sp; = /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.201= 24

=
/var/log/mesos:
root@mesos-slave:/#&= nbsp;
root@mesos-slave:/# = cat /var/log/mesos-fetcher.INFO 
Log file created = at: 2015/12/19 18:16:21
Running on = machine: mesos-slave
Log line format: = [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1219 = 18:16:21.127075 20157 logging.cpp:172] INFO level logging = started!
I1219 = 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: = {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-7= 3d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"= value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-p= racticals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/d= b70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88c= a-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920= cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
I1219 = 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pr= acticals/snoqualmie.csv'
I1219 = 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the = sandbox directory
I1219 = 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pr= acticals/snoqualmie.csv'
I1219 = 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pr= acticals/snoqualmie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv'
I1219 = 18:16:23.221889 20157 fetcher.cpp:454] Fetched 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pr= acticals/snoqualmie.csv' to = '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b= 272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e= 5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.c= sv'
root@mesos-slave:/#&= nbsp;

root@mesos-slave:~# = docker exec -it = mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47= f58970b4 bash
root@mesos-slave:/#&= nbsp;
root@mesos-slave:/#&= nbsp;
root@mesos-slave:/# = ls /mnt/mesos/sandbox/  
snoqualmie.csv stderr stdout
root@mesos-slave:/#&= nbsp;


Best = Regards,
Marica



Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <tim@mesosphere.io> ha = scritto:

HI = Marica,

Did you see the fetcher invoked at all from = the Slave logs? Doesn't seem possible we don't pass down the URI flags = and if the fetcher failed the container launch should have failed = too.

Also another possible situation is that = web UI not really showing the exact content of the sandbox, can you = actually go into the directory and see if the files are = there?

Thanks,

Tim
=

On Fri, = Dec 18, 2015 at 4:23 PM, Marica Antonacci <marica.antonacci@ba.infn.it> = wrote:
Hi = Tim,

looking at the sandbox I can see only the stderr = and stout file (see the attached screenshot). If I remove = =E2=80=94docker_mesos_image (and therefore the executor is run inside = the slave container) the file specified in the URI field is shown in the = sandbox.
Did you verify that the fetcher is called when using = the =E2=80=94docker_mesos_image flag ?

Thanks a = lot for your = feedback
Marica

<Schermata = 2015-12-18 alle 10.15.58.png>

Il = giorno 19/dic/2015, alle ore 00:25, Tim Chen <tim@mesosphere.io> ha = scritto:

Hi = Marica,

It should work as we fetch all the files = before we launch the executor and place them in the sandbox, and we = mount the sandbox into that container as = well.

How did you verify that the file is not = downloaded? 

Tim

On Fri, Dec 18, = 2015 at 5:26 AM, Marica Antonacci <marica.antonacci@ba.infn.it> = wrote:
Hi Grzegorz,

I=E2=80= =99m using this command line for docker run

# = docker run -d MESOS_HOSTNAME=3D<SLAVE_IP> -e = MESOS_IP=3D<SLAVE_IP> -e = MESOS_MASTER=3Dzk://<node-1>:2181,<node-2>:2181,<node-3>= :2181/mesos -e MESOS_CONTAINERIZERS=3Ddocker,mesos -e = MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/var/log = -e MESOS_docker_mesos_image=3Dmesos-slave = -v /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos = --name slave --net host --privileged --pid host = mesos-slave

where mesos-slave is the image = built from the docker file in this repo https://github.com/maricaantonacci/mesos-slave-dev

I have tested successfully the deployment of = dockerized applications through Marathon and dockerized jobs through = Chronos and also the recovery seems to work fine with the flag = docker_mesos_image. What is not working for me is the fetcher: it seems = that when the executor is launched as separate container (thanks to the = flag docker_mesos_image) the information about the URIs to be downloaded = is lost=E2=80=A6I hope someone can help to understand if this a bug or = I=E2=80=99 m missing = something.

Cheers,
Marica

  

Il giorno = 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <gregory90@gmail.com> ha = scritto:

I've tried = to use this flag, but cannot really run any container when this flag is = set. 

pt., = 18.12.2015 o 11:33 u=C5=BCytkownik Marica Antonacci <marica.antonacci@ba.infn.it> = napisa=C5=82:
OK, the problem I spotted is related = to the usage of the flag =E2=80=94docker_mesos_image that allows = the executor to 


--docker_mesos_image=3DVALUEThe docker image used to launch this mesos = slave instance. If an image is specified, the docker containerizer = assumes the slave is running in a docker container, and launches = executors with docker containers in order to recover them when the slave = restarts and recovers.
Has anyone used = this flag and tested the behavior of the = fetcher?

Thank = you
Marica


Il giorno = 18/dic/2015, alle ore 10:38, tommy xiao <xiaods@gmail.com> ha = scritto:

no docker_mesos_image flag in my docker run, =  and the docker image is build by myself.



2015-12-18 17:20 = GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
Yes, = I did check inside the container and the csv file was not downloaded as = shown also by the app details (see the screenshot = below).

Are you running your slave with the = --docker_mesos_image flag? Can you please provide me the docker run = command you are using to run your dockerized slave? =  

Thank you very = much
Marica


<Schermata 2015-12-18 alle = 10.15.58.png>


Il = giorno 18/dic/2015, alle ore 10:00, tommy xiao <xiaods@gmail.com> ha = scritto:

Hi= Marica,

use your test-app json, i can run it = correctly, the csv is truely download by mesos slave. please check = mesos-master:5050 to check the task detail download = files.

you describe the app container why not = found the csv, because the csv is download in slave container's folder, = not in app container. so if you run 

cd $MESOS_SANDBOX;

the folder in app container is default = value: 

MESOS_SANDBOX=3D/mnt/mesos/sandbox<= /span>

but in real world, = the sandbox is in slave container, not in app = container.



2015-12-18 16:11 = GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
Thank you very = much,

I=E2=80=99m using a sample application = definition file, just for testing = purpose:

{ 
 "id": = "test-app",
 "container": {
   "type": = "DOCKER",
   "docker": {
     "image": = "libmesos/ubuntu"
   }
 },
 "cpus": = 1,
 "mem": 512,
 "uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-pr= acticals/snoqualmie.csv" ],
 "cmd": "cd $MESOS_SANDBOX; = ls -latr; while sleep 10; do date -u +%T; = done" 
}

Here is the docker run command = line:

# docker run -d -e = MESOS_HOSTNAME=3D<SLAVE_IP> -e MESOS_IP=3D<SLAVE_IP> -e = MESOS_MASTER=3Dzk://<node-1>:2181,<node-2>:2181,<node-3>= :2181/mesos -e MESOS_CONTAINERIZERS=3Ddocker,mesos = \
                =       -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e = MESOS_LOG_DIR=3D/var/log -e = MESOS_docker_mesos_image=3Dmesos-slave 
    =                   = -v /sys/fs/cgroup:/sys/fs/cgroup -v = /var/run/docker.sock:/var/run/docker.sock --name slave --net host = --privileged --pid host = mesos-slave


As already = mentioned, if I remove the environment variable MESOS_docker_mesos_image = the fetcher works fine and I can see the file snoqualmie.csv inside the = sandbox.

Thank you again! I=E2=80=99m looking = forward to hearing about your outcomes.
Best = regards,
Marica


Il = giorno 18/dic/2015, alle ore 04:51, tommy xiao <xiaods@gmail.com> ha = scritto:

Hi = Marica,

Could you please give a sample marathon json, = i can test it asap.

2015-12-18 5:11 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
No, = using the socket:

-v = /var/run/docker.sock:/var/run/docker.sock

Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xiaods@gmail.com> ha = scritto:

docker in = docker mode?

2015-12-17 19:08 GMT+08:00 Marica Antonacci <marica.antonacci@ba.infn.it>:
Dear = all,

I'm testing the URIs fetching mechanism for both = Marathon applications and Chronos jobs and I have found that if the = slave is running inside a docker container = (using docker_mesos_image startup flag) and you submit the = deployment of a dockerized application or job the fetcher step is not = performed. On the other hand, if I request the deployment of a = non-dockerized application, the URIs are correctly fetched. Moreover, if = I don=E2=80=99t provide the docker_mesos_image flag, the fetcher works = fine again for both dockerized and non-dockerized = applications.

Therefore, it seems that the = information about the URIs gets lost when the dockerized mesos slave = spawns the executor docker container that in turn launches the = application docker container=E2=80=A6Has anyone seen this problem = before? I would like to know if there is a workaround or a = fixing.

Thanks a lot in advance for you = help
Best = Regards,
Marica


----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







--=
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







--
Deshi Xiao
Twitter: = xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







--=
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
<=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.16.png><=E5=B1=8F=E5=B9=95=E5=BF= =AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.30.png><=E5=B1=8F=E5=B9=95=E5=BF= =AB=E7=85=A7 2015-12-18 = =E4=B8=8B=E5=8D=884.55.38.png>

= 
----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




=







-- =
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 





----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 






----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 






----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 





----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 







----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it




= 










----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN - National Institute =
of Nuclear Physics
Via Orabona 4
70126 Bari =
- ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it












--
Best = Regards,
Haosdent Huang

----------------------------------------------------------<=
/font>
Marica ANTONACCI
INFN =
- National Institute of Nuclear Physics
Via =
Orabona 4
70126 Bari - ITALY
Phone +39 080 =
5443244
Skype: marica.antonacci
e-mail marica.antonacci@ba.infn.it









= --Apple-Mail=_A91B2C61-71E0-4E2E-9C67-EAA72A60A480--