Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F295D18E40 for ; Wed, 24 Feb 2016 10:30:03 +0000 (UTC) Received: (qmail 23383 invoked by uid 500); 24 Feb 2016 10:23:24 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 23323 invoked by uid 500); 24 Feb 2016 10:23:24 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 23313 invoked by uid 99); 24 Feb 2016 10:23:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2016 10:23:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 5F90CC04EC for ; Wed, 24 Feb 2016 10:23:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.552 X-Spam-Level: **** X-Spam-Status: No, score=4.552 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, TRACKER_ID=1.102, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id G2q6Q7GDZdxV for ; Wed, 24 Feb 2016 10:23:12 +0000 (UTC) Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 7DF0C5FAD6 for ; Wed, 24 Feb 2016 10:23:11 +0000 (UTC) Received: by mail-io0-f175.google.com with SMTP id 9so30092351iom.1 for ; Wed, 24 Feb 2016 02:23:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=uEpF70v2VrU17MrF9XahC5paKxYx99ASgDSgyuOndyw=; b=oF+VxmWkF0KL1+RR7IGi498lV4xovsr+jCtjzcTolsrY9wkKoLHclfseAiack4nUDp HhhRkmQiNysxjaf/ahRRRBOxUneL//oW9JqKDwl31EwDFP3K55COvKQxCkdB+OqYpCzG zGra7hREDizAd9VtPJuxV6BxTT/LFLwWh+W9ffZq9CK4LsFIeJSHWSbCT3FlqQGw1qx/ +Gzqp20UlC+LnCJDlLt0bQEWWHB4gcO2AAhCA3fDu8lQ6HvNxu+YH06XrEDelx6zqjKH schB6G/bRWCnhZ9neOve7F7/PZvBOs0X3Q4rpz6BsQ350AhUsbYhfolPd9Nf3iCj4ymz fNzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=uEpF70v2VrU17MrF9XahC5paKxYx99ASgDSgyuOndyw=; b=cd+lmv3UdVYzaMq5TFnTizikmvGQHmz0SQbj+Tp1Hf/uLlIql/FWzzoL7/yWbKl23N 4pA8Eh+fEFPW48oWxXl4G2DcpTq9WwkTCvDtBje5EDLunA9OL9TGsYvRwnp+Y+JCnQyi qosI3Cgzjmg5WNkfDgLzRHTh0jFdmIREZdLZ5iBvUGu+Xjfij+hV5vVpand/JRqgBMqj Pi9fHBqxKcoMXAEBxJKkl10YNA+sixLXP1bHxdaAb79ofHqX8kFLLanYIlsHAjk1L7N5 6LPSYIF+nTYd6gn3ypWwEPKAhj5m1WLWDcPrGyos6gvtldRdg970LmMUw3eyT0espPiw 3ENw== X-Gm-Message-State: AG10YOSmNYraxfKogP4MWETmPGerGZqVSA4KLuK4RWjTuhNvtO52zYxN+JWXG6EcVwQOEzd6fqInnA7v68NFlA== MIME-Version: 1.0 X-Received: by 10.107.19.90 with SMTP id b87mr42629979ioj.11.1456309390309; Wed, 24 Feb 2016 02:23:10 -0800 (PST) Received: by 10.36.138.133 with HTTP; Wed, 24 Feb 2016 02:23:10 -0800 (PST) In-Reply-To: References: <1147A296-CB9E-43F8-AB8C-CBF96D125C99@ba.infn.it> <8A1C0ED7-C9E3-4418-A1B2-0528B4FC34F8@ba.infn.it> <91874EDD-9CEC-41C6-A4D4-92301DF1D754@ba.infn.it> <6A68B80A-1485-4836-9776-94163475D977@ba.infn.it> <537DA9F1-F257-419F-999A-47629F007C27@ba.infn.it> <3C823D0B-DF75-4854-82CF-A4B2DC1A742D@mesosphere.io> Date: Wed, 24 Feb 2016 18:23:10 +0800 Message-ID: Subject: Re: Mesos fetcher in dockerized slave From: Shuai Lin To: Timothy Chen Cc: Marica Antonacci , user@mesos.apache.org Content-Type: multipart/alternative; boundary=001a113f3bd4e480f6052c81733f --001a113f3bd4e480f6052c81733f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable ping @Tim, I think this bug also affects https://issues.apache.org/jira/browse/MESOS-4743 . On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin wrote: > The testing of this case requires to build a docker image for mesos-slave= , > so it seems not practical to add a test case for it in the mesos tests. > > Anyway, here is the scripts I use for testing this issue: > https://gist.github.com/lins05/14455e92f37e91fd46ff > > On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin > wrote: > >> Hi Tim, >> >> The review is here: https://reviews.apache.org/r/42390/ , would you >> please take a look? >> >> Regards, >> Shuai >> >> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin wrote= : >> >>> Hi Maria and Tim, >>> >>> I'm setting up a test case for this scenario that would fail, after >>> which I'll begin on fixing it. >>> >>> Is it feasible to include the fixing in the new release? >>> >>> >>> I'm not sure, does 0.27 have an estimated release date now? >>> >>> >>> Regards, >>> Shuai >>> >>> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen wrote: >>> >>>> I can shepherd no problem. >>>> >>>> Tim >>>> >>>> On Dec 25, 2015, at 4:32 PM, Shuai Lin wrote: >>>> >>>> I'll work on it. @Tim could you shepherd it? >>>> >>>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci < >>>> marica.antonacci@ba.infn.it> wrote: >>>> >>>>> Hi Tim and Shuai, >>>>> >>>>> thank you very much for your reply. I have opened a JIRA issue on >>>>> this: https://issues.apache.org/jira/browse/MESOS-4249 >>>>> I hope it will be patched soon :) >>>>> >>>>> Best regards, >>>>> Marica >>>>> >>>>> >>>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen >>>>> ha scritto: >>>>> >>>>> Hi Marica/Shuai, >>>>> >>>>> Sorry haven't been able to spend the time to repro, but looks like >>>>> Shuai confirmed it. >>>>> >>>>> Can one of you file a JIRA? >>>>> >>>>> Thanks! >>>>> >>>>> Tim >>>>> >>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin >>>>> wrote: >>>>> >>>>>> Hi Marica, >>>>>> >>>>>> I can reproduce the problem exactly as you described in the first >>>>>> email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment >>>>>> variable set, the fetcher works just fine; With it, the fetcher step= s seems >>>>>> skipped. This looks like a bug to me. >>>>>> >>>>>> Regards, >>>>>> Shuai >>>>>> >>>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci < >>>>>> marica.antonacci@ba.infn.it> wrote: >>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> I have not solved this issue yet. Please, can anyone run the same >>>>>>> test and let me know if the fetcher is correctly invoked? >>>>>>> The test is really simple, just try to start a dockerized app (see >>>>>>> json definition file below) through marathon on a mesos slave runni= ng in a >>>>>>> docker container started with the option =E2=80=94docker_mesos_imag= e=3D>>>>>> image>. >>>>>>> I would appreciate very much any feedback. >>>>>>> >>>>>>> Sample Marathon app: >>>>>>> { >>>>>>> "id": "test-app", >>>>>>> "container": { >>>>>>> "type": "DOCKER", >>>>>>> "docker": { >>>>>>> "image": "libmesos/ubuntu" >>>>>>> } >>>>>>> }, >>>>>>> "cpus": 1, >>>>>>> "mem": 512, >>>>>>> "uris": [ " >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/sn= oqualmie.csv" >>>>>>> ], >>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u >>>>>>> +%T; done" >>>>>>> } >>>>>>> >>>>>>> Docker run command to start dockerized mesos slave: >>>>>>> >>>>>>> # docker run -d MESOS_HOSTNAME=3D -e MESOS_IP=3D -e >>>>>>> MESOS_MASTER=3Dzk://:2181,:2181,:2181/mesos= -e >>>>>>> MESOS_CONTAINERIZERS=3Ddocker,mesos >>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/= var/log -e >>>>>>> MESOS_docker_mesos_image=3Dmesos-slave -v /sys/fs/cgroup:/sys/fs/cg= roup -v >>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos >>>>>>> --name slave --net host --privileged --pid host mesos-slave >>>>>>> >>>>>>> Thank you very much in advance! >>>>>>> Best regards, >>>>>>> Marica >>>>>>> >>>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci < >>>>>>> marica.antonacci@ba.infn.it> ha scritto: >>>>>>> >>>>>>> Dear Tim, >>>>>>> >>>>>>> I have collected some information from my test environment, startin= g >>>>>>> the slave container with and without the =E2=80=94docker_mesos_imag= e startup flag. >>>>>>> Please let me know if you need further input. Thank you very much f= or your >>>>>>> support! >>>>>>> >>>>>>> Using the flag =E2=80=94docker_mesos_image: >>>>>>> >>>>>>> root@mesos-slave:~# docker ps >>>>>>> CONTAINER ID IMAGE COMMAND >>>>>>> CREATED STATUS PORTS NAMES >>>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 >>>>>>> minutes ago Up 2 minutes >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95= d2-f3370214c84d >>>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 >>>>>>> minutes ago Up 2 minutes >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95= d2-f3370214c84d.executor >>>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 >>>>>>> minutes ago Up 3 minutes slave >>>>>>> >>>>>>> *root@mesos-slave:~# docker logs slave* >>>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a6= 7a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>>>>> to user 'root' >>>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.= 1; >>>>>>> mem(*):32 in work directory >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a6= 7a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container >>>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework >>>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000' >>>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 >>>>>>> to >>>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/fra= meworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398a= f9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pi= ds/forked.pid' >>>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for >>>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framewo= rk >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating >>>>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources pas= sed to >>>>>>> update is identical to existing resources >>>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor >>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update >>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received >>>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea= 552) for >>>>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] >>>>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID: >>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update >>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209 >>>>>>> :5050 >>>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement >>>>>>> for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea= 58ea552) >>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@ >>>>>>> 90.147.170.246:39381 >>>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received >>>>>>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea5= 8ea552) >>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] >>>>>>> Checkpointing ACK for status update TASK_RUNNING (UUID: >>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>>> >>>>>>> root@mesos-slave:~# *docker logs >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95= d2-f3370214c84d.executor* >>>>>>> --container=3D"mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f5= 9b-cc1a-4081-95d2-f3370214c84d" >>>>>>> --docker=3D"docker" --docker_socket=3D"/var/run/docker.sock" --help= =3D"false" >>>>>>> --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec= /mesos" >>>>>>> --logbufsecs=3D"0" --logging_level=3D"INFO" >>>>>>> --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" >>>>>>> --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73= d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executor= s/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d= 2-f3370214c84d" >>>>>>> --stop_timeout=3D"0ns" >>>>>>> --container=3D"mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f5= 9b-cc1a-4081-95d2-f3370214c84d" >>>>>>> --docker=3D"docker" --docker_socket=3D"/var/run/docker.sock" --help= =3D"false" >>>>>>> --initialize_driver_logging=3D"true" --launcher_dir=3D"/usr/libexec= /mesos" >>>>>>> --logbufsecs=3D"0" --logging_level=3D"INFO" >>>>>>> --mapped_directory=3D"/mnt/mesos/sandbox" --quiet=3D"false" >>>>>>> --sandbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73= d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executor= s/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d= 2-f3370214c84d" >>>>>>> --stop_timeout=3D"0ns" >>>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0 >>>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on >>>>>>> slave db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>>>>> Registered docker executor on 90.147.170.246 >>>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>>> >>>>>>> root@mesos-slave:~# *docker exec -it >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95= d2-f3370214c84d.executor >>>>>>> bash* >>>>>>> root@mesos-slave:/# ls -R /tmp/ >>>>>>> hsperfdata_root/ mesos/ >>>>>>> root@mesos-slave:/# ls -R /tmp/mesos/ >>>>>>> /tmp/mesos/: >>>>>>> *slaves* >>>>>>> >>>>>>> /tmp/mesos/slaves: >>>>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0* >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0: >>>>>>> *frameworks* >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framework= s: >>>>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framework= s/246b272b-d649-47c0-88ca-6b1ff35f437a-0000: >>>>>>> *executors* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framework= s/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors: >>>>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framework= s/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67= a-11e5-b1cf-fa163e920cd0: >>>>>>> *runs* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framework= s/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67= a-11e5-b1cf-fa163e920cd0/runs: >>>>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d* >>>>>>> >>>>>>> >>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framework= s/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67= a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d: >>>>>>> stderr stdout >>>>>>> >>>>>>> >>>>>>> root@mesos-slave:~# *docker exec -it >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95= d2-f3370214c84d >>>>>>> bash* >>>>>>> root@mesos-slave:/# env >>>>>>> HOSTNAME=3Dmesos-slave >>>>>>> HOST=3D90.147.170.246 >>>>>>> PORT0=3D31220 >>>>>>> PORT_10000=3D31220 >>>>>>> MESOS_TASK_ID=3Dtest-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>>> PATH=3D/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>>>> PWD=3D/ >>>>>>> PORTS=3D31220 >>>>>>> >>>>>>> MESOS_CONTAINER_NAME=3Dmesos-db70e09f-f39d-491c-8480-73d9858c140b-S= 0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>>> SHLVL=3D1 >>>>>>> HOME=3D/ >>>>>>> MARATHON_APP_ID=3D/test-app >>>>>>> MARATHON_APP_DOCKER_IMAGE=3Dlibmesos/ubuntu >>>>>>> MARATHON_APP_VERSION=3D2015-12-19T18:03:37.542Z >>>>>>> PORT=3D31220 >>>>>>> MESOS_SANDBOX=3D/mnt/mesos/sandbox >>>>>>> _=3D/usr/bin/env >>>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX >>>>>>> /mnt/mesos/sandbox: >>>>>>> stderr stdout >>>>>>> >>>>>>> >>>>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>>>> */var/log/mesos-slave.INFO* >>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182= 512.20647 >>>>>>> >>>>>>> /var/log/mesos: >>>>>>> >>>>>>> >>>>>>> Disabling the flag =E2=80=94docker_mesos_image the fetcher is calle= d, log is >>>>>>> created, the file is downloaded in the sandbox: >>>>>>> >>>>>>> root@mesos-slave:~# docker exec -it slave bash >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>>>> */var/log/mesos-fetcher.INFO* >>>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-1= 81621.20157 >>>>>>> */var/log/mesos-slave.INFO* >>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181= 612.20124 >>>>>>> >>>>>>> /var/log/mesos: >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO >>>>>>> Log file created at: 2015/12/19 18:16:21 >>>>>>> Running on machine: mesos-slave >>>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg >>>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging >>>>>>> started! >>>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: >>>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c= -8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":f= alse,"value":"http:\/\/ >>>>>>> www.stat.cmu.edu >>>>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"= sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c1= 40b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/t= est-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6= -3f47f58970b4"} >>>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/sn= oqualmie.csv' >>>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into >>>>>>> the sandbox directory >>>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/sn= oqualmie.csv' >>>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource >>>>>>> from ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/sn= oqualmie.csv' >>>>>>> to >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a6= 7c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqual= mie.csv' >>>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched ' >>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/sn= oqualmie.csv' >>>>>>> to >>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/framewor= ks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a6= 7c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqual= mie.csv' >>>>>>> root@mesos-slave:/# >>>>>>> >>>>>>> root@mesos-slave:~# docker exec -it >>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8b= c6-3f47f58970b4 >>>>>>> bash >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# >>>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/ >>>>>>> snoqualmie.csv stderr stdout >>>>>>> root@mesos-slave:/# >>>>>>> >>>>>>> >>>>>>> Best Regards, >>>>>>> Marica >>>>>>> >>>>>>> >>>>>>> >>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen >>>>>>> ha scritto: >>>>>>> >>>>>>> HI Marica, >>>>>>> >>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't >>>>>>> seem possible we don't pass down the URI flags and if the fetcher f= ailed >>>>>>> the container launch should have failed too. >>>>>>> >>>>>>> Also another possible situation is that web UI not really showing >>>>>>> the exact content of the sandbox, can you actually go into the dire= ctory >>>>>>> and see if the files are there? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci < >>>>>>> marica.antonacci@ba.infn.it> wrote: >>>>>>> >>>>>>>> Hi Tim, >>>>>>>> >>>>>>>> looking at the sandbox I can see only the stderr and stout file >>>>>>>> (see the attached screenshot). If I remove =E2=80=94docker_mesos_i= mage (and >>>>>>>> therefore the executor is run inside the slave container) the file >>>>>>>> specified in the URI field is shown in the sandbox. >>>>>>>> Did you verify that the fetcher is called when using the >>>>>>>> =E2=80=94docker_mesos_image flag ? >>>>>>>> >>>>>>>> Thanks a lot for your feedback >>>>>>>> Marica >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen >>>>>>>> ha scritto: >>>>>>>> >>>>>>>> Hi Marica, >>>>>>>> >>>>>>>> It should work as we fetch all the files before we launch the >>>>>>>> executor and place them in the sandbox, and we mount the sandbox i= nto that >>>>>>>> container as well. >>>>>>>> >>>>>>>> How did you verify that the file is not downloaded? >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci < >>>>>>>> marica.antonacci@ba.infn.it> wrote: >>>>>>>> >>>>>>>>> Hi Grzegorz, >>>>>>>>> >>>>>>>>> I=E2=80=99m using this command line for docker run >>>>>>>>> >>>>>>>>> # docker run -d MESOS_HOSTNAME=3D -e MESOS_IP=3D >>>>>>>>> -e MESOS_MASTER=3Dzk://:2181,:2181,:2181/= mesos >>>>>>>>> -e MESOS_CONTAINERIZERS=3Ddocker,mesos -e >>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR=3D/v= ar/log -e >>>>>>>>> MESOS_docker_mesos_image=3Dmesos-slave -v /sys/fs/cgroup:/sys/fs/= cgroup -v >>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/meso= s --name >>>>>>>>> slave --net host --privileged --pid host mesos-slave >>>>>>>>> >>>>>>>>> where mesos-slave is the image built from the docker file in this >>>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev >>>>>>>>> >>>>>>>>> I have tested successfully the deployment of dockerized >>>>>>>>> applications through Marathon and dockerized jobs through Chronos= and also >>>>>>>>> the recovery seems to work fine with the flag docker_mesos_image.= What is >>>>>>>>> not working for me is the fetcher: it seems that when the executo= r is >>>>>>>>> launched as separate container (thanks to the flag docker_mesos_i= mage) the >>>>>>>>> information about the URIs to be downloaded is lost=E2=80=A6I hop= e someone can help >>>>>>>>> to understand if this a bug or I=E2=80=99 m missing something. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Marica >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk < >>>>>>>>> gregory90@gmail.com> ha scritto: >>>>>>>>> >>>>>>>>> I've tried to use this flag, but cannot really run any container >>>>>>>>> when this flag is set. >>>>>>>>> I've raised this issue here: >>>>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html = and >>>>>>>>> here: >>>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecom= ment-155364351 but >>>>>>>>> sadly no one was able to help me... >>>>>>>>> >>>>>>>>> pt., 18.12.2015 o 11:33 u=C5=BCytkownik Marica Antonacci < >>>>>>>>> marica.antonacci@ba.infn.it> napisa=C5=82: >>>>>>>>> >>>>>>>>>> OK, the problem I spotted is related to the usage of the >>>>>>>>>> flag =E2=80=94docker_mesos_image that allows the executor to >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --docker_mesos_image=3DVALUE The docker image used to launch thi= s >>>>>>>>>> mesos slave instance. If an image is specified, the docker conta= inerizer >>>>>>>>>> assumes the slave is running in a docker container, and launches= executors >>>>>>>>>> with docker containers in order to recover them when the slave r= estarts and >>>>>>>>>> recovers. >>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher= ? >>>>>>>>>> >>>>>>>>>> Thank you >>>>>>>>>> Marica >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao < >>>>>>>>>> xiaods@gmail.com> ha scritto: >>>>>>>>>> >>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker >>>>>>>>>> image is build by myself. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci < >>>>>>>>>> marica.antonacci@ba.infn.it>: >>>>>>>>>> >>>>>>>>>> Yes, I did check inside the container and the csv file was not >>>>>>>>>>> downloaded as shown also by the app details (see the screenshot= below). >>>>>>>>>>> >>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag? >>>>>>>>>>> Can you please provide me the docker run command you are using = to run your >>>>>>>>>>> dockerized slave? >>>>>>>>>>> >>>>>>>>>>> Thank you very much >>>>>>>>>>> >>>>>>>>>> Marica >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao < >>>>>>>>>>> xiaods@gmail.com> ha scritto: >>>>>>>>>>> >>>>>>>>>>> Hi Marica, >>>>>>>>>>> >>>>>>>>>>> use your test-app json, i can run it correctly, the csv is >>>>>>>>>>> truely download by mesos slave. please check mesos-master:5050 = to check the >>>>>>>>>>> task detail download files. >>>>>>>>>>> >>>>>>>>>>> you describe the app container why not found the csv, because >>>>>>>>>>> the csv is download in slave container's folder, not in app con= tainer. so >>>>>>>>>>> if you run >>>>>>>>>>> >>>>>>>>>>> cd $MESOS_SANDBOX; >>>>>>>>>>> >>>>>>>>>>> the folder in app container is default value: >>>>>>>>>>> >>>>>>>>>>> MESOS_SANDBOX=3D/mnt/mesos/sandbox >>>>>>>>>>> but in real world, the sandbox is in slave container, not in ap= p >>>>>>>>>>> container. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci < >>>>>>>>>>> marica.antonacci@ba.infn.it>: >>>>>>>>>>> >>>>>>>>>>>> Thank you very much, >>>>>>>>>>>> >>>>>>>>>>>> I=E2=80=99m using a sample application definition file, just f= or >>>>>>>>>>>> testing purpose: >>>>>>>>>>>> >>>>>>>>>>>> { >>>>>>>>>>>> "id": "test-app", >>>>>>>>>>>> "container": { >>>>>>>>>>>> "type": "DOCKER", >>>>>>>>>>>> "docker": { >>>>>>>>>>>> "image": "libmesos/ubuntu" >>>>>>>>>>>> } >>>>>>>>>>>> }, >>>>>>>>>>>> "cpus": 1, >>>>>>>>>>>> "mem": 512, >>>>>>>>>>>> *"uris": [ >>>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practic= als/snoqualmie.csv >>>>>>>>>>>> " >>>>>>>>>>>> ],* >>>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date >>>>>>>>>>>> -u +%T; done" >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> Here is the docker run command line: >>>>>>>>>>>> >>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=3D -e >>>>>>>>>>>> MESOS_IP=3D -e >>>>>>>>>>>> MESOS_MASTER=3Dzk://:2181,:2181,:2181/= mesos >>>>>>>>>>>> -e MESOS_CONTAINERIZERS=3Ddocker,mesos \ >>>>>>>>>>>> -e >>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_LOG_DIR= =3D/var/log -e >>>>>>>>>>>> MESOS_docker_mesos_image=3Dmesos-slave >>>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v >>>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net h= ost >>>>>>>>>>>> --privileged --pid host mesos-slave >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> As already mentioned, if I remove the environment variable >>>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see = the file >>>>>>>>>>>> snoqualmie.csv inside the sandbox. >>>>>>>>>>>> >>>>>>>>>>>> Thank you again! I=E2=80=99m looking forward to hearing about = your >>>>>>>>>>>> outcomes. >>>>>>>>>>>> Best regards, >>>>>>>>>>>> Marica >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao < >>>>>>>>>>>> xiaods@gmail.com> ha scritto: >>>>>>>>>>>> >>>>>>>>>>>> Hi Marica, >>>>>>>>>>>> >>>>>>>>>>>> Could you please give a sample marathon json, i can test it >>>>>>>>>>>> asap. >>>>>>>>>>>> >>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci < >>>>>>>>>>>> marica.antonacci@ba.infn.it>: >>>>>>>>>>>> >>>>>>>>>>>>> No, using the socket: >>>>>>>>>>>>> >>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao < >>>>>>>>>>>>> xiaods@gmail.com> ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>> docker in docker mode? >>>>>>>>>>>>> >>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci < >>>>>>>>>>>>> marica.antonacci@ba.infn.it>: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dear all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon >>>>>>>>>>>>>> applications and Chronos jobs and I have found that if the s= lave is running >>>>>>>>>>>>>> inside a docker container (using *docker_mesos_image* >>>>>>>>>>>>>> startup flag) and you submit the deployment of a dockerized = application or >>>>>>>>>>>>>> job the fetcher step is not performed. On the other hand, if= I request the >>>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are cor= rectly fetched. >>>>>>>>>>>>>> Moreover, if I don=E2=80=99t provide the docker_mesos_image = flag, the fetcher works >>>>>>>>>>>>>> fine again for both dockerized and non-dockerized applicatio= ns. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets >>>>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor doc= ker container >>>>>>>>>>>>>> that in turn launches the application docker container=E2=80= =A6Has anyone seen this >>>>>>>>>>>>>> problem before? I would like to know if there is a workaroun= d or a fixing. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks a lot in advance for you help >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Marica >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>> >>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Deshi Xiao >>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>> <=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 =E4=B8=8B=E5= =8D=884.55.16.png><=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 =E4=B8= =8B=E5=8D=884.55.30.png><=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 >>>>>>>>>>> 2015-12-18 =E4=B8=8B=E5=8D=884.55.38.png> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>> >>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>> Via Orabona 4 >>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Deshi Xiao >>>>>>>>>> Twitter: xds2000 >>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>> >>>>>>>>>> Marica ANTONACCI >>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>> Via Orabona 4 >>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>> Skype: marica.antonacci >>>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> ---------------------------------------------------------- >>>>>>>>> >>>>>>>>> Marica ANTONACCI >>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>> Via Orabona 4 >>>>>>>>> 70126 Bari - ITALY >>>>>>>>> Phone +39 080 5443244 >>>>>>>>> Skype: marica.antonacci >>>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> ---------------------------------------------------------- >>>>>>>> >>>>>>>> Marica ANTONACCI >>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>> Via Orabona 4 >>>>>>>> 70126 Bari - ITALY >>>>>>>> Phone +39 080 5443244 >>>>>>>> Skype: marica.antonacci >>>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------- >>>>>>> >>>>>>> Marica ANTONACCI >>>>>>> INFN - National Institute of Nuclear Physics >>>>>>> Via Orabona 4 >>>>>>> 70126 Bari - ITALY >>>>>>> Phone +39 080 5443244 >>>>>>> Skype: marica.antonacci >>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------- >>>>>>> >>>>>>> Marica ANTONACCI >>>>>>> INFN - National Institute of Nuclear Physics >>>>>>> Via Orabona 4 >>>>>>> 70126 Bari - ITALY >>>>>>> Phone +39 080 5443244 >>>>>>> Skype: marica.antonacci >>>>>>> e-mail marica.antonacci@ba.infn.it >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> ---------------------------------------------------------- >>>>> >>>>> Marica ANTONACCI >>>>> INFN - National Institute of Nuclear Physics >>>>> Via Orabona 4 >>>>> 70126 Bari - ITALY >>>>> Phone +39 080 5443244 >>>>> Skype: marica.antonacci >>>>> e-mail marica.antonacci@ba.infn.it >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >> > --001a113f3bd4e480f6052c81733f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
ping @Tim, I think this bug also affects=C2=A0https://issues.apache.org/= jira/browse/MESOS-4743 .



On Wed, Jan 20, 2016 at 10:20 PM, Shuai = Lin <linshuai2012@gmail.com> wrote:
The testing of this case requires to build a docker imag= e for mesos-slave, so it seems not practical to add a test case for it in t= he mesos tests.=C2=A0

Anyway, here is the scripts I use = for testing this issue: https://gist.github.com/lins05/14455e92f37e9= 1fd46ff

On Wed, Jan 20, 2016 at 10:30 AM, S= huai Lin <linshuai2012@gmail.com> wrote:
Hi Tim,

The review is here:=C2=A0= https://r= eviews.apache.org/r/42390/ , would you please take a look?
Regards,
Shuai

On Sat, Jan 9, 2016 at 9:42 AM, S= huai Lin <linshuai2012@gmail.com> wrote:
Hi Maria and Tim,=C2=A0

I'm s= etting up a test case for this scenario that would fail, after which I'= ll begin on fixing it.=C2=A0

Is it feasible to include the fixing in the new release?

I'm not sure, does 0.27 have an estimated releas= e date now?


Regards,
Shuai

On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen = <tim@mesosphere.i= o> wrote:
I can sh= epherd no problem.

T= im

On Dec 25, 2015, at 4:32 PM, S= huai Lin <li= nshuai2012@gmail.com> wrote:

=
I'll work on it. @Tim could you shepherd it?

On Sat, Dec 26, = 2015 at 2:49 AM, Marica Antonacci <marica.antonacci@ba.infn.it> wrote:
Hi Tim and Shuai,

thank you very much for your reply. = I have opened a JIRA issue on this:=C2=A0https://issues.apache.org/jira= /browse/MESOS-4249
I hope it will be patched soon :)=C2=A0

Best regards,
Marica

Il giorno 24/dic/2015, alle ore 17:54, Tim Chen &= lt;tim@mesosphere.io= > ha scritto:

Hi= Marica/Shuai,

Sorry haven't been able to spend the = time to repro, but looks like Shuai confirmed it.

= Can one of you file a JIRA?

Thanks!

=
Tim

On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <<= a href=3D"mailto:linshuai2012@gmail.com" target=3D"_blank">linshuai2012@gma= il.com> wrote:
Hi Marica= ,

I can reproduce the problem exactly as you described i= n the first email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` enviro= nment variable set, the fetcher works just fine; With it, the fetcher steps= seems skipped. This looks like a bug to me.

Regar= ds,
Shuai

On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <= span dir=3D"ltr"><marica.antonacci@ba.infn.it> wrote:
Dear all,

I h= ave not solved this issue yet. Please, can anyone run the same test and let= me know if the fetcher is correctly invoked?=C2=A0
The test is r= eally simple, just try to start a dockerized app (see json definition file = below) through marathon on a mesos slave running in a docker container star= ted with the option =E2=80=94docker_mesos_image=3D<mesos slave image>= .
I would appreciate very much any feedback.=C2=A0

=
Sample Marathon app:
{=C2=A0
=C2=A0"id&q= uot;: "test-app",
=C2=A0"container": {
=C2=A0 =C2= =A0"type": "DOCKER",
=C2=A0 =C2=A0"docker"= : {
=C2=A0 =C2=A0 =C2=A0"image": "libmesos/ubuntu"=C2=A0 =C2=A0}
=C2=A0},
=C2=A0"cpus": 1,
=C2=A0"m= em": 512,
=C2=A0"uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practical= s/snoqualmie.csv" ],
=C2=A0"cmd": "cd $MESOS_SAN= DBOX; ls -latr; while sleep 10; do date -u +%T; done"=C2=A0
}
=

Docker run command to start dockerized mesos sla= ve:

# docker run -d MESOS_HOSTNAME=3D<SLA= VE_IP> -e MESOS_IP=3D<SLAVE_IP> -e MESOS_MASTER=3Dzk://<node-1&= gt;:2181,<node-2>:2181,<node-3>:2181/mesos -e MESOS_CONTAINERIZ= ERS=3Ddocker,mesos -e=C2=A0MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e M= ESOS_LOG_DIR=3D/var/log -e MESOS_docker_mesos_image=3Dmesos-slave -v /sys/f= s/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tm= p/mesos:/tmp/mesos --name=C2=A0slave --net host --privileged --pid host mes= os-slave

Thank you very much in advance!
Best regards,
Marica

Il giorno 19= /dic/2015, alle ore 19:32, Marica Antonacci <marica.antonacci@ba.infn.it> h= a scritto:

Dear Tim,

I have collected some inform= ation from my test environment, starting the slave container with and witho= ut the =E2=80=94docker_mesos_image startup flag. Please let me know if you = need further input. Thank you very much for your support!

Using the flag =E2=80=94docker_mesos_image:

root@mesos-slave:~# dock= er ps
CONTAINER ID=C2=A0 =C2=A0 =C2=A0 =C2=A0 IMAGE =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 COMMAND=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 CREATED =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = STATUS=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 PORTS =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 NAMES
b30cea22a07c=C2=A0 =C2=A0 =C2=A0 =C2=A0 libme= sos/ubuntu =C2=A0 =C2=A0 "/bin/sh -c 'cd $MESO" =C2=A0 2 minu= tes ago =C2=A0 =C2=A0 =C2=A0 Up 2 minutes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mesos-db70e= 09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
da9c78ec5727= =C2=A0 =C2=A0 =C2=A0 =C2=A0 mesos-slave =C2=A0 =C2=A0 =C2=A0 =C2=A0 "/= bin/sh -c '/usr/lib" =C2=A0 2 minutes ago =C2=A0 =C2=A0 =C2=A0 Up = 2 minutes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mesos-db70e09f-f39d-491c-8480-73d9858c140b-= S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
150f78fbf327=C2=A0 =C2=A0 =C2=A0 =C2= =A0 mesos-slave =C2=A0 =C2=A0 =C2=A0 =C2=A0 "/entrypoint.sh /usr/"= ; =C2=A0 3 minutes ago =C2=A0 =C2=A0 =C2=A0 Up 3 minutes=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 slave

root@mesos-slave:~# docker logs slave
I1219 18:03:3= 8.308544 19476 slave.cpp:1294] Got assigned task test-app.d4398af9-a67a-11e= 5-b1cf-fa163e920cd0 for framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000=
I1219 18:03:38.314268 19476 slave.cpp:1410]= Launching task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework= 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I= 1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown '/tmp/mesos/s= laves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0= -88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e92= 0cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' to user 'root'<= /div>
I1219 18:03:38.327221 19476 slave.cpp:4999] = Launching executor test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framewo= rk 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; me= m(*):32 in work directory '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73= d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executor= s/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d= 2-f3370214c84d'
I1219 18:03:38.330817 19= 476 slave.cpp:1628] Queuing task 'test-app.d4398af9-a67a-11e5-b1cf-fa16= 3e920cd0' for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920c= d0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:38.335907 19480 docker.cpp:762] Starting con= tainer 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task 'test-ap= p.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor 'test-app.d43= 98af9-a67a-11e5-b1cf-fa163e920cd0') of framework '246b272b-d649-47c= 0-88ca-6b1ff35f437a-0000'
I1219 18:03:40= .084027 19482 docker.cpp:386] Checkpointing pid 19590 to '/tmp/mesos/me= ta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-= 47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa16= 3e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
I1219 18:03:40.189172 19479 slave.cpp:2405] Got= registration for executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920c= d0' of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executo= r(1)@90.147.170.= 246:39381
I1219 18:03:40.191141 19479 do= cker.cpp:1012] Ignoring updating container 'd965f59b-cc1a-4081-95d2-f33= 70214c84d' with resources passed to update is identical to existing res= ources
I1219 18:03:40.192512 19479 slave.cpp= :1793] Sending queued task 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920c= d0' to executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0'= of framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@90.147.170.246:39381=
I1219 18:03:42.425606 19483 slave.cpp:2= 762] Handling status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5= ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framew= ork 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@90.147.170.246:39381
I1219 18:03:42.426415 19483 status_update_manag= er.cpp:322] Received status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-a= fcb-0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of= framework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpo= inting UPDATE for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb= -0f5ea58ea552) for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of fr= amework 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update TAS= K_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app.d4= 398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-6b1= ff35f437a-0000 to master@90.147.102.209:5050
I1219 18:03= :42.521852 19483 slave.cpp:3011] Sending acknowledgement for status update = TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-app= .d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88ca-= 6b1ff35f437a-0000 to executor(1)@90.147.170.246:39381
I1= 219 18:03:42.576373 19481 status_update_manager.cpp:394] Received status up= date acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task = test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47= c0-88ca-6b1ff35f437a-0000
I1219 18:03:42.577= 263 19481 status_update_manager.cpp:826] Checkpointing ACK for status updat= e TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task test-a= pp.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework 246b272b-d649-47c0-88c= a-6b1ff35f437a-0000

root@mesos-slave:~# docker logs mesos-db70e09f-f39d-491c-8480-73d9858= c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
--container= =3D"mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-9= 5d2-f3370214c84d" --docker=3D"docker" --docker_socket=3D&quo= t;/var/run/docker.sock" --help=3D"false" --initialize_driver= _logging=3D"true" --launcher_dir=3D"/usr/libexec/mesos"= --logbufsecs=3D"0" --logging_level=3D"INFO" --mapped_d= irectory=3D"/mnt/mesos/sandbox" --quiet=3D"false" --san= dbox_directory=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c14= 0b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-a= pp.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f33702= 14c84d" --stop_timeout=3D"0ns"
--container=3D"mesos-= db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84= d" --docker=3D"docker" --docker_socket=3D"/var/run/dock= er.sock" --help=3D"false" --initialize_driver_logging=3D&quo= t;true" --launcher_dir=3D"/usr/libexec/mesos" --logbufsecs= =3D"0" --logging_level=3D"INFO" --mapped_directory=3D&q= uot;/mnt/mesos/sandbox" --quiet=3D"false" --sandbox_director= y=3D"/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewo= rks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a= 67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" = --stop_timeout=3D"0ns"
I1219 18:03:40.177598=C2=A0=C2=A0 =C2= =A0=C2=A06 exec.cpp:136] Version: 0.26.0
I1219 18:03:40.192060=C2=A0 =C2= =A0=C2=A010 exec.cpp:210] Executor registered on slave db70e09f-f39d-491c-8= 480-73d9858c140b-S0
Registered docker executor on 90.147.170.246
Star= ting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0

root@mesos-slave:~# <= b>docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc= 1a-4081-95d2-f3370214c84d.executor bash
root@mesos-slave:/# ls -R /tmp/
hsperfdata_root/ meso= s/=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
root@mesos-slave:/# ls -R /tmp/mesos/
/tmp/mesos/:
slaves

/tmp/mesos/slaves:
db70e09f-f39d-491c-848= 0-73d9858c140b-S0

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b= -S0:
frameworks

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-7= 3d9858c140b-S0/frameworks:
246b272b-d649-47c0-88ca-6b1ff35f437= a-0000

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/framewo= rks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
executors

/t= mp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b= -d649-47c0-88ca-6b1ff35f437a-0000/executors:
test-app.d4398af9= -a67a-11e5-b1cf-fa163e920cd0

/tmp/mesos/slaves/db70e09f-f39d-491c-8480-7= 3d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executo= rs/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
runs

/tmp= /mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d= 649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-= fa163e920cd0/runs:
d965f59b-cc1a-4081-95d2-f3370214c84d

/tm= p/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-= d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf= -fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
stderr=C2=A0 stdout
=C2=A0
root= @mesos-slave:~# docker exec -it mesos-db70e09f-f39d-491c-8480-73d9858c14= 0b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d bash
root@mesos-slave:/# env
HOSTNAME=3Dmesos-slav= e
HOST=3D90= .147.170.246
PORT0=3D31220
PORT_10000=3D31220
MESOS_TASK_ID=3Dtest-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0<= /div>
PATH=3D/usr= /local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=3D/
PORTS=3D31220
MESOS_CONTAINER_NAME=3Dmesos-= db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84= d
SHLVL=3D1=
HOME=3D/
MARATHON_APP= _ID=3D/test-app
MARATHON_APP_DOCKER_IMAGE=3Dlibmesos/ubuntu
MARATHON_APP_VERSION=3D2015-12-19T18:= 03:37.542Z
= PORT=3D31220
MESOS_SANDBOX=3D/mnt/mesos/sandbox
_=3D/usr/bin/env
root@mesos-slave:/# ls -R $MESOS_SANDBOX
/mnt/mesos/sa= ndbox:
stde= rr stdout

<= /div>

root@mesos-slave:/# ls /var/log/mesos*
= /var/log/mesos-slave.INFO=C2=A0 /var/log/mesos-slave.mesos-sl= ave.invalid-user.log.INFO.20151219-182512.20647

/var/log/mesos:


Disabling the flag =E2=80=94docker_meso= s_image the fetcher is called, log is created, the file is downloaded in th= e sandbox:

root@mesos-slave:~# docker exec -it slave bash
root@mesos-slave:/#= =C2=A0
root= @mesos-slave:/#=C2=A0
root@mesos-slave:/#=C2=A0
root@mesos-slave:/# ls /var/log/mesos*
/var/log/mesos-fetcher.INFO=C2=A0 /var/log/m= esos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157=C2=A0 = /var/log/mesos-slave.INFO=C2=A0 /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-18= 1612.20124

/var/log/mesos:
root@mesos-slave:/#=C2=A0
root@mesos-slave:/# cat /var/log/mesos-fetche= r.INFO=C2=A0
Log file created at: 2015/12/19 18:16:21
Running on machine: mesos-slave
Log line format: [IWEF= ]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1219 18:16:21.127075 20157 logging.cpp:= 172] INFO level logging started!
I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher = Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e= 09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action&q= uot;:"BYPASS_CACHE","uri":{"extract":false,&q= uot;value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practica= ls\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/meso= s\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d6= 49-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf= -fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
I1219 18:16:21.129547= 20157 fetcher.cpp:377] Fetching URI 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/sno= qualmie.csv'
I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly in= to the sandbox directory
I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI = 9;http://www.stat.cmu.edu/~cshalizi= /402/lectures/16-glm-practicals/snoqualmie.csv'
I1219 18:16:21.129636 20157 fe= tcher.cpp:132] Downloading resource from 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practical= s/snoqualmie.csv' to '/tmp/mesos/slaves/db70e09f-f39d-491c-8480= -73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/execu= tors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-= 8bc6-3f47f58970b4/snoqualmie.csv'
I1219 18:16:23.221889 20157 fetcher.cpp:454] Fet= ched 'http://www.stat.cmu.edu/~= cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' to '/tm= p/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-= d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf= -fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'=
root@mesos= -slave:/#=C2=A0

root@mesos-slave:~# docker exec -it mesos-db= 70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 = bash
root@m= esos-slave:/#=C2=A0
root@mesos-slave:/#=C2=A0
root@mesos-slave:/# ls /mnt/mesos/sandbox/ =C2=A0
snoqualmie.c= sv stderr stdout
root@mesos-slave:/#=C2=A0


Best Regards,
Marica


Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <tim@mesosphere.io&= gt; ha scritto:

HI Mari= ca,

Did you see the fetcher invoked at all from the Slav= e logs? Doesn't seem possible we don't pass down the URI flags and = if the fetcher failed the container launch should have failed too.

Also another possible situation is that web UI not really = showing the exact content of the sandbox, can you actually go into the dire= ctory and see if the files are there?

Thanks,

Tim

On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <marica.antonacci@ba.infn.it> wrote:
Hi Tim,

looking at = the sandbox I can see only the stderr and stout file (see the attached scre= enshot). If I remove =E2=80=94docker_mesos_image (and therefore the executo= r is run inside the slave container) the file specified in the URI field is= shown in the sandbox.
Did you verify that the fetcher is called = when using the =E2=80=94docker_mesos_image flag ?

= Thanks a lot for your feedback
Marica

<Schermata 2015-12-18 alle 10.15.58.png>

Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <tim@mesosphere.io> ha scritto= :

Hi Marica,

It should work as we fetch all the files before we launch the exec= utor and place them in the sandbox, and we mount the sandbox into that cont= ainer as well.

How did you verify that the file is= not downloaded?=C2=A0

Tim

On Fri, Dec 18, 2015 at 5:2= 6 AM, Marica Antonacci <marica.antonacci@ba.infn.it> wrote:
Hi=C2=A0G= rzegorz,

I=E2=80=99m using this command line for docker = run

# docker run -d MESOS_HOSTNAME=3D<SLAVE_IP&= gt; -e MESOS_IP=3D<SLAVE_IP> -e MESOS_MASTER=3Dzk://<node-1>:21= 81,<node-2>:2181,<node-3>:2181/mesos -e=C2=A0MESOS_CONTAINERIZE= RS=3Ddocker,mesos -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins -e MESOS_L= OG_DIR=3D/var/log -e MESOS_docker_mesos_image=3Dmesos-slave -v=C2=A0/sys/fs= /cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/docker.sock -v /tmp= /mesos:/tmp/mesos --name slave --net host --privileged --pid host mesos-sla= ve

where mesos-slave is the image built from the d= ocker file in this repo=C2=A0https://github.com/maricaantonacci/mesos= -slave-dev

I have tested successfully the depl= oyment of dockerized applications through Marathon and dockerized jobs thro= ugh Chronos and also the recovery seems to work fine with the flag docker_m= esos_image. What is not working for me is the fetcher: it seems that when t= he executor is launched as separate container (thanks to the flag docker_me= sos_image) the information about the URIs to be downloaded is lost=E2=80=A6= I hope someone can help to understand if this a bug or I=E2=80=99 m missing= something.

Cheers,
Marica

=C2=A0=C2=A0

Il giorno 18/dic/2015= , alle ore 12:11, Grzegorz Graczyk <gregory90@gmail.com> ha scritto:

I've tried to use this flag, but = cannot really run any container when this flag is set.=C2=A0
I've r= aised this issue here:=C2=A0https://www.mail-archive.com/user@mesos.apache.org/msg04975= .html=C2=A0and here:=C2=A0https://gi= thub.com/mesosphere/docker-containers/issues/6#issuecomment-155364351= =C2=A0but sadly no one was able to help me...

pt., 18.12.2015 o 11:33=C2=A0u=C5=BCytkownik = Marica Antonacci <marica.antonacci@ba.infn.it> napisa=C5=82:
OK, the problem I spo= tted is related to the usage of the flag=C2=A0=E2=80=94docker_mesos_image t= hat allows the executor to=C2=A0


--docke= r_mesos_image=3DVALUEThe docker image used = to launch this mesos slave instance. If an image is specified, the docker c= ontainerizer assumes the slave is running in a docker container, and launch= es executors with docker containers in order to recover them when the slave= restarts and recovers.
Has anyone used this = flag and tested the behavior of the fetcher?

Thank you
Marica


Il giorno 18/= dic/2015, alle ore 10:38, tommy xiao <xiaods@gmail.com> ha scritto:

no=C2=A0do= cker_mesos_image flag in my docker run, =C2=A0and the docker image is build= by myself.



=
2015-12-18 17:20 GMT+08:00 Marica = Antonacci <marica.antonacci@ba.infn.it>:
=
Yes, I did check inside the con= tainer and the csv file was not downloaded as shown also by the app details= (see the screenshot below).

Are you running your slave = with the --docker_mesos_image flag? Can you please provide me the docker ru= n command you are using to run your dockerized slave? =C2=A0

=
Thank you very much
Marica


= <Schermata 2015-12-18 alle 10.15.58.png>

<= div>
Il giorno 18/dic/2015, alle ore 10:00, tommy xiao &l= t;xiaods@gmail.com> ha scritto:


2015-12-18 16:11 GMT+08:00 Mar= ica Antonacci <marica.antonacci@ba.infn.it>:
Thank you very much= ,

I=E2=80=99m using a sample application definitio= n file, just for testing purpose:

{=C2=A0
=C2= =A0"id": "test-app",
=C2=A0"container": {<= br>=C2=A0 =C2=A0"type": "DOCKER",
=C2=A0 =C2=A0"= ;docker": {
=C2=A0 =C2=A0 =C2=A0"image": "libmesos/u= buntu"
=C2=A0 =C2=A0}
=C2=A0},
=C2=A0"cpus": 1,
= =C2=A0"mem": 512,
=C2=A0"uris": [ "http://www.stat.cmu.edu/~cshalizi/402/lectures/= 16-glm-practicals/snoqualmie.csv" ],
=C2=A0"cmd":= "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; done&qu= ot;=C2=A0
}

Here is the docker run command line:
=

# docker run -d -e MESOS_HOSTNAME=3D<SLAVE_IP> -e= MESOS_IP=3D<SLAVE_IP> -e MESOS_MASTER=3Dzk://<node-1>:2181,<= ;node-2>:2181,<node-3>:2181/mesos -e=C2=A0MESOS_CONTAINERIZERS=3Dd= ocker,mesos \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=3D5mins = -e MESOS_LOG_DIR=3D/var/log -e MESOS_docker_mesos_image=3Dmesos-slave=C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 -v=C2=A0/sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock= :/var/run/docker.sock --name slave --net host --privileged --pid host mesos= -slave


As already mentioned, if I r= emove the environment variable MESOS_docker_mesos_image the fetcher works f= ine and I can see the file snoqualmie.csv inside the sandbox.
Thank you again! I=E2=80=99m looking forward to hearing about y= our outcomes.
Best regards,
Marica

<= /div>
Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xiaods@gmail.com>= ; ha scritto:

Hi Marica= ,

Could you please give a sample marathon json, i can te= st it asap.

2015-12-18 5:11 GMT+08:00 Marica Antonacci <marica.antonac= ci@ba.infn.it>:
No, using the socket:

-v /var/run/docker.sock= :/var/run/docker.sock


Il giorno= 17/dic/2015, alle ore 18:07, tommy xiao <xiaods@gmail.com> ha scritto:

docker in docker mode?

2015-12-17 19:08 GMT+08:00= Marica Antonacci <marica.antonacci@ba.infn.it>:
Dear all,

I'm testing the URIs fetching mechanism for both Marathon appl= ications and Chronos jobs and I have found that if the slave is running ins= ide a docker container (using=C2=A0docker_mesos_image startup flag) = and you submit the deployment of a dockerized application or job the fetche= r step is not performed. On the other hand, if I request the deployment of = a non-dockerized application, the URIs are correctly fetched. Moreover, if = I don=E2=80=99t provide the docker_mesos_image flag, the fetcher works fine= again for both dockerized and non-dockerized applications.

<= /div>
Therefore, it seems that the information about the URIs gets lost= when the dockerized mesos slave spawns the executor docker container that = in turn launches the application docker container=E2=80=A6Has anyone seen t= his problem before? I would like to know if there is a workaround or a fixi= ng.

Thanks a lot in advance for you help
Best Regards,
Marica


----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it












-- =
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it












--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it












-- =
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
<=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 =E4=B8=8B= =E5=8D=884.55.16.png><=E5=B1=8F=E5=B9=95=E5=BF=AB=E7=85= =A7 2015-12-18 =E4=B8=8B=E5=8D=884.55.30.png><=E5=B1=8F= =E5=B9=95=E5=BF=AB=E7=85=A7 2015-12-18 =E4=B8=8B=E5=8D=884.55.38.png>

----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it












--
Deshi Xiao
Twitte= r: xds2000
E-mail: xiaods(AT)gmail.com

----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it










----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it











----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it











----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it










----------------------------------------------------------=

Marica=
=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0Nuclear PhysicsVia Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone +39=C2=A00=
80 5443244
Skype:=C2=A0marica.antonacci
e-mail=C2=A0marica.antonacci@ba.inf=
n.it












-----------------------------------------=
-----------------
Marica=C2=A0ANTONACCI
INFN -=C2=A0National Institute of=C2=A0=
Nuclear Physics
Via Orabona=C2=A04
70126 Bari -=C2=A0ITALY
Phone <=
a href=3D"tel:%2B39%C2%A0080%205443244" value=3D"+390805443244" target=3D"_=
blank">+39=C2=A0080 5443244
Skype:=C2=A0marica.antonacci
e-mail=
=C2=A0mari=
ca.antonacci@ba.infn.it






<= br>






--001a113f3bd4e480f6052c81733f--