mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuai Lin <linshuai2...@gmail.com>
Subject Re: Mesos fetcher in dockerized slave
Date Wed, 24 Feb 2016 10:23:10 GMT
ping @Tim, I think this bug also affects
https://issues.apache.org/jira/browse/MESOS-4743 .



On Wed, Jan 20, 2016 at 10:20 PM, Shuai Lin <linshuai2012@gmail.com> wrote:

> The testing of this case requires to build a docker image for mesos-slave,
> so it seems not practical to add a test case for it in the mesos tests.
>
> Anyway, here is the scripts I use for testing this issue:
> https://gist.github.com/lins05/14455e92f37e91fd46ff
>
> On Wed, Jan 20, 2016 at 10:30 AM, Shuai Lin <linshuai2012@gmail.com>
> wrote:
>
>> Hi Tim,
>>
>> The review is here: https://reviews.apache.org/r/42390/ , would you
>> please take a look?
>>
>> Regards,
>> Shuai
>>
>> On Sat, Jan 9, 2016 at 9:42 AM, Shuai Lin <linshuai2012@gmail.com> wrote:
>>
>>> Hi Maria and Tim,
>>>
>>> I'm setting up a test case for this scenario that would fail, after
>>> which I'll begin on fixing it.
>>>
>>> Is it feasible to include the fixing in the new release?
>>>
>>>
>>> I'm not sure, does 0.27 have an estimated release date now?
>>>
>>>
>>> Regards,
>>> Shuai
>>>
>>> On Sat, Jan 9, 2016 at 1:11 AM, Timothy Chen <tim@mesosphere.io> wrote:
>>>
>>>> I can shepherd no problem.
>>>>
>>>> Tim
>>>>
>>>> On Dec 25, 2015, at 4:32 PM, Shuai Lin <linshuai2012@gmail.com> wrote:
>>>>
>>>> I'll work on it. @Tim could you shepherd it?
>>>>
>>>> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci <
>>>> marica.antonacci@ba.infn.it> wrote:
>>>>
>>>>> Hi Tim and Shuai,
>>>>>
>>>>> thank you very much for your reply. I have opened a JIRA issue on
>>>>> this: https://issues.apache.org/jira/browse/MESOS-4249
>>>>> I hope it will be patched soon :)
>>>>>
>>>>> Best regards,
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <tim@mesosphere.io>
>>>>> ha scritto:
>>>>>
>>>>> Hi Marica/Shuai,
>>>>>
>>>>> Sorry haven't been able to spend the time to repro, but looks like
>>>>> Shuai confirmed it.
>>>>>
>>>>> Can one of you file a JIRA?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Tim
>>>>>
>>>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <linshuai2012@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> I can reproduce the problem exactly as you described in the first
>>>>>> email of this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment
>>>>>> variable set, the fetcher works just fine; With it, the fetcher steps seems
>>>>>> skipped. This looks like a bug to me.
>>>>>>
>>>>>> Regards,
>>>>>> Shuai
>>>>>>
>>>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I have not solved this issue yet. Please, can anyone run the same
>>>>>>> test and let me know if the fetcher is correctly invoked?
>>>>>>> The test is really simple, just try to start a dockerized app (see
>>>>>>> json definition file below) through marathon on a mesos slave running in a
>>>>>>> docker container started with the option —docker_mesos_image=<mesos slave
>>>>>>> image>.
>>>>>>> I would appreciate very much any feedback.
>>>>>>>
>>>>>>> Sample Marathon app:
>>>>>>> {
>>>>>>>  "id": "test-app",
>>>>>>>  "container": {
>>>>>>>    "type": "DOCKER",
>>>>>>>    "docker": {
>>>>>>>      "image": "libmesos/ubuntu"
>>>>>>>    }
>>>>>>>  },
>>>>>>>  "cpus": 1,
>>>>>>>  "mem": 512,
>>>>>>>  "uris": [ "
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"
>>>>>>> ],
>>>>>>>  "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>> +%T; done"
>>>>>>> }
>>>>>>>
>>>>>>> Docker run command to start dockerized mesos slave:
>>>>>>>
>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>>>>>>> MESOS_CONTAINERIZERS=docker,mesos
>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>>>>>>> --name slave --net host --privileged --pid host mesos-slave
>>>>>>>
>>>>>>> Thank you very much in advance!
>>>>>>> Best regards,
>>>>>>> Marica
>>>>>>>
>>>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it> ha scritto:
>>>>>>>
>>>>>>> Dear Tim,
>>>>>>>
>>>>>>> I have collected some information from my test environment, starting
>>>>>>> the slave container with and without the —docker_mesos_image startup flag.
>>>>>>> Please let me know if you need further input. Thank you very much for your
>>>>>>> support!
>>>>>>>
>>>>>>> Using the flag —docker_mesos_image:
>>>>>>>
>>>>>>> root@mesos-slave:~# docker ps
>>>>>>> CONTAINER ID        IMAGE               COMMAND
>>>>>>> CREATED             STATUS              PORTS               NAMES
>>>>>>> b30cea22a07c        libmesos/ubuntu     "/bin/sh -c 'cd $MESO"   2
>>>>>>> minutes ago       Up 2 minutes
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>> da9c78ec5727        mesos-slave         "/bin/sh -c '/usr/lib"   2
>>>>>>> minutes ago       Up 2 minutes
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>>> 150f78fbf327        mesos-slave         "/entrypoint.sh /usr/"   3
>>>>>>> minutes ago       Up 3 minutes                            slave
>>>>>>>
>>>>>>> *root@mesos-slave:~# docker logs slave*
>>>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>>> to user 'root'
>>>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>>>>>>> mem(*):32 in work directory
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>>>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>>>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>>>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>>>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590
>>>>>>> to
>>>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>>>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for
>>>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating
>>>>>>> container 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to
>>>>>>> update is identical to existing resources
>>>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>>>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>>>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>>>>>>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826]
>>>>>>> Checkpointing UPDATE for status update TASK_RUNNING (UUID:
>>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>>>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209
>>>>>>> :5050
>>>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement
>>>>>>> for status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>>>>>>> 90.147.170.246:39381
>>>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>>>>>>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>>>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826]
>>>>>>> Checkpointing ACK for status update TASK_RUNNING (UUID:
>>>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>>>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>>>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>>>>>>
>>>>>>> root@mesos-slave:~# *docker logs
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --stop_timeout="0ns"
>>>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>>>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>>>>>>> --logbufsecs="0" --logging_level="INFO"
>>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>>>>>>> --stop_timeout="0ns"
>>>>>>> I1219 18:03:40.177598     6 exec.cpp:136] Version: 0.26.0
>>>>>>> I1219 18:03:40.192060    10 exec.cpp:210] Executor registered on
>>>>>>> slave db70e09f-f39d-491c-8480-73d9858c140b-S0
>>>>>>> Registered docker executor on 90.147.170.246
>>>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>>
>>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>>>>>>> bash*
>>>>>>> root@mesos-slave:/# ls -R /tmp/
>>>>>>> hsperfdata_root/ mesos/
>>>>>>> root@mesos-slave:/# ls -R /tmp/mesos/
>>>>>>> /tmp/mesos/:
>>>>>>> *slaves*
>>>>>>>
>>>>>>> /tmp/mesos/slaves:
>>>>>>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>>>>>>> *frameworks*
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>>>>>>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>>>>>>> *executors*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>>>>>>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>>>>>>> *runs*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>>>>>>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>>>>>>
>>>>>>>
>>>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>>>>>>> stderr  stdout
>>>>>>>
>>>>>>>
>>>>>>> root@mesos-slave:~# *docker exec -it
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>> bash*
>>>>>>> root@mesos-slave:/# env
>>>>>>> HOSTNAME=mesos-slave
>>>>>>> HOST=90.147.170.246
>>>>>>> PORT0=31220
>>>>>>> PORT_10000=31220
>>>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>>>>>> PWD=/
>>>>>>> PORTS=31220
>>>>>>>
>>>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>>>>>>> SHLVL=1
>>>>>>> HOME=/
>>>>>>> MARATHON_APP_ID=/test-app
>>>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>>>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>>>>>>> PORT=31220
>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>> _=/usr/bin/env
>>>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>>>>>>> /mnt/mesos/sandbox:
>>>>>>> stderr stdout
>>>>>>>
>>>>>>>
>>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>>> */var/log/mesos-slave.INFO*
>>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>>>>>>
>>>>>>> /var/log/mesos:
>>>>>>>
>>>>>>>
>>>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is
>>>>>>> created, the file is downloaded in the sandbox:
>>>>>>>
>>>>>>> root@mesos-slave:~# docker exec -it slave bash
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/# ls /var/log/mesos*
>>>>>>> */var/log/mesos-fetcher.INFO*
>>>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>>>>>>> */var/log/mesos-slave.INFO*
>>>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>>>>>>
>>>>>>> /var/log/mesos:
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>>>>>>> Log file created at: 2015/12/19 18:16:21
>>>>>>> Running on machine: mesos-slave
>>>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging
>>>>>>> started!
>>>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>>>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>>>>>>> www.stat.cmu.edu
>>>>>>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>>>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into
>>>>>>> the sandbox directory
>>>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource
>>>>>>> from '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> to
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>>>>>>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>>>>>>> to
>>>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>>>>>>> root@mesos-slave:/#
>>>>>>>
>>>>>>> root@mesos-slave:~# docker exec -it
>>>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>>>>>>> bash
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/#
>>>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>>>>>>> snoqualmie.csv stderr stdout
>>>>>>> root@mesos-slave:/#
>>>>>>>
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <tim@mesosphere.io>
>>>>>>> ha scritto:
>>>>>>>
>>>>>>> HI Marica,
>>>>>>>
>>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't
>>>>>>> seem possible we don't pass down the URI flags and if the fetcher failed
>>>>>>> the container launch should have failed too.
>>>>>>>
>>>>>>> Also another possible situation is that web UI not really showing
>>>>>>> the exact content of the sandbox, can you actually go into the directory
>>>>>>> and see if the files are there?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Tim
>>>>>>>
>>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>
>>>>>>>> Hi Tim,
>>>>>>>>
>>>>>>>> looking at the sandbox I can see only the stderr and stout file
>>>>>>>> (see the attached screenshot). If I remove —docker_mesos_image (and
>>>>>>>> therefore the executor is run inside the slave container) the file
>>>>>>>> specified in the URI field is shown in the sandbox.
>>>>>>>> Did you verify that the fetcher is called when using the
>>>>>>>> —docker_mesos_image flag ?
>>>>>>>>
>>>>>>>> Thanks a lot for your feedback
>>>>>>>> Marica
>>>>>>>>
>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>
>>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <tim@mesosphere.io>
>>>>>>>> ha scritto:
>>>>>>>>
>>>>>>>> Hi Marica,
>>>>>>>>
>>>>>>>> It should work as we fetch all the files before we launch the
>>>>>>>> executor and place them in the sandbox, and we mount the sandbox into that
>>>>>>>> container as well.
>>>>>>>>
>>>>>>>> How did you verify that the file is not downloaded?
>>>>>>>>
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>>>>>>> marica.antonacci@ba.infn.it> wrote:
>>>>>>>>
>>>>>>>>> Hi Grzegorz,
>>>>>>>>>
>>>>>>>>> I’m using this command line for docker run
>>>>>>>>>
>>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP>
>>>>>>>>> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>>>>>>> slave --net host --privileged --pid host mesos-slave
>>>>>>>>>
>>>>>>>>> where mesos-slave is the image built from the docker file in this
>>>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev
>>>>>>>>>
>>>>>>>>> I have tested successfully the deployment of dockerized
>>>>>>>>> applications through Marathon and dockerized jobs through Chronos and also
>>>>>>>>> the recovery seems to work fine with the flag docker_mesos_image. What is
>>>>>>>>> not working for me is the fetcher: it seems that when the executor is
>>>>>>>>> launched as separate container (thanks to the flag docker_mesos_image) the
>>>>>>>>> information about the URIs to be downloaded is lost…I hope someone can help
>>>>>>>>> to understand if this a bug or I’ m missing something.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>>>>>>> gregory90@gmail.com> ha scritto:
>>>>>>>>>
>>>>>>>>> I've tried to use this flag, but cannot really run any container
>>>>>>>>> when this flag is set.
>>>>>>>>> I've raised this issue here:
>>>>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>>>>>>> here:
>>>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 but
>>>>>>>>> sadly no one was able to help me...
>>>>>>>>>
>>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>>>>>>> marica.antonacci@ba.infn.it> napisał:
>>>>>>>>>
>>>>>>>>>> OK, the problem I spotted is related to the usage of the
>>>>>>>>>> flag —docker_mesos_image that allows the executor to
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this
>>>>>>>>>> mesos slave instance. If an image is specified, the docker containerizer
>>>>>>>>>> assumes the slave is running in a docker container, and launches executors
>>>>>>>>>> with docker containers in order to recover them when the slave restarts and
>>>>>>>>>> recovers.
>>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>>>>>>
>>>>>>>>>> Thank you
>>>>>>>>>> Marica
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <
>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>
>>>>>>>>>> no docker_mesos_image flag in my docker run,  and the docker
>>>>>>>>>> image is build by myself.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>
>>>>>>>>>> Yes, I did check inside the container and the csv file was not
>>>>>>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>>>>>>
>>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag?
>>>>>>>>>>> Can you please provide me the docker run command you are using to run your
>>>>>>>>>>> dockerized slave?
>>>>>>>>>>>
>>>>>>>>>>> Thank you very much
>>>>>>>>>>>
>>>>>>>>>> Marica
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <
>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>
>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>
>>>>>>>>>>> use your test-app json, i can run it correctly, the csv is
>>>>>>>>>>> truely download by mesos slave. please check mesos-master:5050 to check the
>>>>>>>>>>> task detail download files.
>>>>>>>>>>>
>>>>>>>>>>> you describe the app container why not found the csv, because
>>>>>>>>>>> the csv is download in slave container's folder, not in app container. so
>>>>>>>>>>> if you run
>>>>>>>>>>>
>>>>>>>>>>> cd $MESOS_SANDBOX;
>>>>>>>>>>>
>>>>>>>>>>> the folder in app container is default value:
>>>>>>>>>>>
>>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>>>>>>> but in real world, the sandbox is in slave container, not in app
>>>>>>>>>>> container.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you very much,
>>>>>>>>>>>>
>>>>>>>>>>>> I’m using a sample application definition file, just for
>>>>>>>>>>>> testing purpose:
>>>>>>>>>>>>
>>>>>>>>>>>> {
>>>>>>>>>>>>  "id": "test-app",
>>>>>>>>>>>>  "container": {
>>>>>>>>>>>>    "type": "DOCKER",
>>>>>>>>>>>>    "docker": {
>>>>>>>>>>>>      "image": "libmesos/ubuntu"
>>>>>>>>>>>>    }
>>>>>>>>>>>>  },
>>>>>>>>>>>>  "cpus": 1,
>>>>>>>>>>>>  "mem": 512,
>>>>>>>>>>>>  *"uris": [
>>>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>>>>>>> ],*
>>>>>>>>>>>>  "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date
>>>>>>>>>>>> -u +%T; done"
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> Here is the docker run command line:
>>>>>>>>>>>>
>>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e
>>>>>>>>>>>> MESOS_IP=<SLAVE_IP> -e
>>>>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>>>>>>>                       -e
>>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>>>>>>>>>> MESOS_docker_mesos_image=mesos-slave
>>>>>>>>>>>>                       -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>>>>>>> --privileged --pid host mesos-slave
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you again! I’m looking forward to hearing about your
>>>>>>>>>>>> outcomes.
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Marica
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <
>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Marica,
>>>>>>>>>>>>
>>>>>>>>>>>> Could you please give a sample marathon json, i can test it
>>>>>>>>>>>> asap.
>>>>>>>>>>>>
>>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>
>>>>>>>>>>>>> No, using the socket:
>>>>>>>>>>>>>
>>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <
>>>>>>>>>>>>> xiaods@gmail.com> ha scritto:
>>>>>>>>>>>>>
>>>>>>>>>>>>> docker in docker mode?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>>>>>>> marica.antonacci@ba.infn.it>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>>>>>>> applications and Chronos jobs and I have found that if the slave is running
>>>>>>>>>>>>>> inside a docker container (using *docker_mesos_image*
>>>>>>>>>>>>>> startup flag) and you submit the deployment of a dockerized application or
>>>>>>>>>>>>>> job the fetcher step is not performed. On the other hand, if I request the
>>>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly fetched.
>>>>>>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher works
>>>>>>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Therefore, it seems that the information about the URIs gets
>>>>>>>>>>>>>> lost when the dockerized mesos slave spawns the executor docker container
>>>>>>>>>>>>>> that in turn launches the application docker container…Has anyone seen this
>>>>>>>>>>>>>> problem before? I would like to know if there is a workaround or a fixing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Marica
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Deshi Xiao
>>>>>>>>>>> Twitter: xds2000
>>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>>> Via Orabona 4
>>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Deshi Xiao
>>>>>>>>>> Twitter: xds2000
>>>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>
>>>>>>>>>> Marica ANTONACCI
>>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>>> Via Orabona 4
>>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>>> Phone +39 080 5443244
>>>>>>>>>> Skype: marica.antonacci
>>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antonacci@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message