mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From haosdent <haosd...@gmail.com>
Subject Re: Can't start docker container when SSL_ENABLED is on.
Date Sun, 01 Nov 2015 09:40:33 GMT
@Xiaodong I create a ticket to trace this
https://issues.apache.org/jira/browse/MESOS-3815 and post a patch in it.
Feel free to review and test it together. Thank you!

On Sun, Nov 1, 2015 at 4:54 PM, haosdent <haosdent@gmail.com> wrote:

> Hi, @Xiaodong I could reproduce your problem in my testing today. A
> quickly workaround is adding environment variables when you launch slave.
>
> ```
> ./bin/mesos-slave.sh xxxx --containerizers=docker,mesos
> --executor_environment_variables='{"SSL_KEY_FILE": "/tmp/server.key",
> "SSL_CERT_FILE": "/tmp/ssl.chain.crt", "SSL_ENABLED": "true"}''
> ```
>
> As you see above, pass the ssl env to docker-executor through specifying
> --executor_environment_variables when starting. So far it works well for
> me. Anyway I would submit a patch later to fix the docker environment
> variables passing. After that, you could launch slave without
> executor_environment_variables flag.
>
> On Sat, Oct 31, 2015 at 2:56 PM, Tim Chen <tim@mesosphere.io> wrote:
>
>> Hi Xiaodong,
>>
>> If you follow the reviewboard you'll see that the fix is not correct, I
>> believe Jojy will be posting a new patch.
>>
>> Tim
>>
>> On Fri, Oct 30, 2015 at 6:58 PM, Xiaodong Zhang <xdzhang@alauda.io>
>> wrote:
>>
>>> it is still not working!
>>>
>>> Only if I remove SSL_ENABLED from envs before I start the slave it works
>>> well.
>>>
>>> I applied the patch in version 0.24.1. And rebuild it with `--enable-libevent
>>> --enable-ssl` 。
>>>
>>> 发件人: Xiaodong Zhang <xdzhang@alauda.io>
>>> 日期: 2015年10月31日 星期六 上午7:45
>>>
>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>
>>> Thanks Jojy.
>>>
>>> I will patch this in version 0.24.1, and rebuild it. I will let you know
>>> if it work well after I finish testing.
>>>
>>> 发件人: Jojy Varghese <jojy@mesosphere.io>
>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>> 日期: 2015年10月31日 星期六 上午12:45
>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>
>>> Thanks Xiaodong.
>>>
>>> Based on the hypothesis that the container process launched with
>>> SSL_ENABLED in environment is the problem, I have created a patch
>>> https://reviews.apache.org/r/39818/.  This might be a quick and dirty
>>> was to test the hypothesis. Would it be possible for you to test again
>>> after applying the patch?
>>>
>>> -Jojy
>>>
>>>
>>>
>>> On Oct 30, 2015, at 8:29 AM, Xiaodong Zhang <xdzhang@alauda.io> wrote:
>>>
>>> Thanks @Jojy
>>>
>>>
>>>
>>> Flags at startup: --appc_store_dir="/tmp/mesos/store/appc"
>>> --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false"
>>> --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup"
>>> --cgroups_limit_swap="false" --cgroups_root="mesos"
>>> --container_disk_watch_interval="15secs" --containerizers="docker,mesos"
>>> --credential="/etc/mesos-slave-auth" --default_role="*"
>>> --disk_watch_interval="1mins" --docker="/usr/bin/docker"
>>> --docker_kill_orphans="true" --docker_remove_delay="6hrs"
>>> --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
>>> --enforce_container_disk_quota="false"
>>> --executor_registration_timeout="1hrs"
>>> --executor_shutdown_grace_period="5secs"
>>> --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB"
>>> --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1"
>>> --hadoop_home="" --help="false" --initialize_driver_logging="true"
>>> --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos"
>>> --log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO"
>>> --master="
>>> zk://172.31.43.77:2181,172.31.44.2:2181,172.31.36.91:2181/mesos"
>>> --oversubscribed_resources_interval="15secs" --perf_duration="10secs"
>>> --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
>>> --quiet="false" --recover="reconnect" --recovery_timeout="15mins"
>>> --registration_backoff_factor="1secs"
>>> --resource_monitoring_interval="1secs" --revocable_cpu_low_priority="true"
>>> --sandbox_directory="/mnt/mesos/sandbox" --strict="true"
>>> --switch_user="true" --version="false" --work_dir="/tmp/mesos"
>>>
>>> 发件人: Jojy Varghese <jojy@mesosphere.io>
>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>> 日期: 2015年10月30日 星期五 下午11:17
>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>
>>> Hi Xiaodong
>>>   This might be because the executor inherits the SSL environment
>>> variables of slave and thus expects SSL key password to launch. Could you
>>> please add the part of the slave logs that says "Flags at startup” so that
>>> we can have more information?
>>>
>>> thanks
>>> Jojy
>>>
>>>
>>> On Oct 29, 2015, at 8:55 PM, Xiaodong Zhang <xdzhang@alauda.io> wrote:
>>>
>>> Thanks a lot !~ @haosent
>>>
>>> 发件人: haosdent <haosdent@gmail.com>
>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>> 日期: 2015年10月30日 星期五 上午11:45
>>> 至: user <user@mesos.apache.org>
>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>
>>> Hi, @Xiaodong I interested in your problem. But recently days I don't
>>> have enough time to try reproduce your problem. I think I could try to dig
>>> your problem at this Sunday and give you feedback.
>>>
>>> On Fri, Oct 30, 2015 at 11:30 AM, Xiaodong Zhang <xdzhang@alauda.io>
>>> wrote:
>>>
>>>> Anybody know about this?
>>>>
>>>> 发件人: Xiaodong Zhang <xdzhang@alauda.io>
>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>> 日期: 2015年10月29日 星期四 下午7:38
>>>>
>>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>>
>>>> I think it is easy to reproduce this error.
>>>>
>>>> Start master with env:
>>>>
>>>> SSL_SUPPORT_DOWNGRADE
>>>> SSL_ENABLED
>>>> SSL_KEY_FILE
>>>> SSL_CERT_FILE
>>>>
>>>> Start slave with env:
>>>>
>>>> SSL_ENABLED
>>>> SSL_KEY_FILE
>>>> SSL_CERT_FILE
>>>> LIBPROCESS_ADVERTISE_IP
>>>>
>>>>
>>>> Then run a docker task via marathon.
>>>>
>>>> 发件人: Xiaodong Zhang <xdzhang@alauda.io>
>>>> 日期: 2015年10月29日 星期四 下午3:09
>>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>>
>>>> So now, mesos task work well but docker task doesn’t.
>>>>
>>>> 发件人: Xiaodong Zhang <xdzhang@alauda.io>
>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>> 日期: 2015年10月29日 星期四 下午2:08
>>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>>
>>>> I run a task by marathon:
>>>>
>>>> {
>>>>     "id": "basic-0",
>>>>     "cmd": "while [ true ] ; do echo 'Hello Marathon' ; sleep 5 ; done",
>>>>     "cpus": 0.1,
>>>>     "mem": 10.0,
>>>>     "instances": 1}
>>>>
>>>>
>>>> It works well.
>>>>
>>>> <742629F2-78E8-43F2-9015-F3D22720826B.png>
>>>>
>>>> Docker task can pull image but can’t run as I mentioned.
>>>>
>>>> My docker version 1.5.0
>>>>
>>>> 发件人: Tim Chen <tim@mesosphere.io>
>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>> 日期: 2015年10月29日 星期四 下午1:48
>>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>>
>>>> Does running a task without docker container (Mesos containerizer)
>>>> works with ssl in your environment?
>>>>
>>>> Tim
>>>>
>>>> On Wed, Oct 28, 2015 at 10:19 PM, Xiaodong Zhang <xdzhang@alauda.io>
>>>> wrote:
>>>>
>>>>> Thanks a lot. I find the log file in slave.
>>>>>
>>>>> One of the task:
>>>>>
>>>>> Stdout:
>>>>>
>>>>> --container="mesos-20151029-043755-3549436724-5050-5674-S0.e2c2580f-8082-4f17-b0cc-4e32e040d444"
>>>>> --docker="/home/ubuntu/luna/bin/docker" --help="false"
>>>>> --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO"
>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>> --sandbox_directory="/tmp/mesos/slaves/20151029-043755-3549436724-5050-5674-S0/frameworks/20151029-043755-3549436724-5050-5674-0000/executors/e4a3bed5-64e6-4970-8bb1-df6404656a48.e3a20f3b-7dfb-11e5-b57b-0247b493b22f/runs/e2c2580f-8082-4f17-b0cc-4e32e040d444"
>>>>> --stop_timeout="0ns"
>>>>> --container="mesos-20151029-043755-3549436724-5050-5674-S0.e2c2580f-8082-4f17-b0cc-4e32e040d444"
>>>>> --docker="/home/ubuntu/luna/bin/docker" --help="false"
>>>>> --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO"
>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>>>>> --sandbox_directory="/tmp/mesos/slaves/20151029-043755-3549436724-5050-5674-S0/frameworks/20151029-043755-3549436724-5050-5674-0000/executors/e4a3bed5-64e6-4970-8bb1-df6404656a48.e3a20f3b-7dfb-11e5-b57b-0247b493b22f/runs/e2c2580f-8082-4f17-b0cc-4e32e040d444"
>>>>> --stop_timeout="0ns"
>>>>> Shutting down
>>>>>
>>>>> Stderr:
>>>>>
>>>>> I1029 05:14:06.529364 27862 fetcher.cpp:414] Fetcher Info:
>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/20151029-043755-3549436724-5050-5674-S0","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"file:\/\/\/etc\/.dockercfg"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/20151029-043755-3549436724-5050-5674-S0\/frameworks\/20151029-043755-3549436724-5050-5674-0000\/executors\/e4a3bed5-64e6-4970-8bb1-df6404656a48.e3a20f3b-7dfb-11e5-b57b-0247b493b22f\/runs\/e2c2580f-8082-4f17-b0cc-4e32e040d444"}
>>>>> I1029 05:14:06.530562 27862 fetcher.cpp:369] Fetching URI '
>>>>> file:///etc/.dockercfg'
>>>>> I1029 05:14:06.530580 27862 fetcher.cpp:243] Fetching directly into
>>>>> the sandbox directory
>>>>> I1029 05:14:06.530594 27862 fetcher.cpp:180] Fetching URI '
>>>>> file:///etc/.dockercfg'
>>>>> I1029 05:14:06.530609 27862 fetcher.cpp:160] Copying resource with
>>>>> command:cp '/etc/.dockercfg'
>>>>> '/tmp/mesos/slaves/20151029-043755-3549436724-5050-5674-S0/frameworks/20151029-043755-3549436724-5050-5674-0000/executors/e4a3bed5-64e6-4970-8bb1-df6404656a48.e3a20f3b-7dfb-11e5-b57b-0247b493b22f/runs/e2c2580f-8082-4f17-b0cc-4e32e040d444/.dockercfg'
>>>>> I1029 05:14:06.532165 27862 fetcher.cpp:446] Fetched '
>>>>> file:///etc/.dockercfg' to
>>>>> '/tmp/mesos/slaves/20151029-043755-3549436724-5050-5674-S0/frameworks/20151029-043755-3549436724-5050-5674-0000/executors/e4a3bed5-64e6-4970-8bb1-df6404656a48.e3a20f3b-7dfb-11e5-b57b-0247b493b22f/runs/e2c2580f-8082-4f17-b0cc-4e32e040d444/.dockercfg'
>>>>> I1029 05:14:07.782054 27955 exec.cpp:133] Version: 0.24.1
>>>>> I1029 05:14:07.785039 27963 exec.cpp:462] Slave exited ... shutting
>>>>> down
>>>>> E1029 05:14:07.785158 27964 socket.hpp:174] Shutdown failed on fd=7:
>>>>> Transport endpoint is not connected [107]
>>>>>
>>>>> 发件人: haosdent <haosdent@gmail.com>
>>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>> 日期: 2015年10月29日 星期四 下午1:13
>>>>>
>>>>> 至: user <user@mesos.apache.org>
>>>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>>>
>>>>> <5185_02_04.png>
>>>>> <5185_02_07.png>
>>>>> ​
>>>>> I capture how I find tasks log in my local webui, could you find the
>>>>> stderr and stdout for your tasks according above screenshots?
>>>>> ​
>>>>>
>>>>> On Thu, Oct 29, 2015 at 1:07 PM, Xiaodong Zhang <xdzhang@alauda.io>
>>>>> wrote:
>>>>>
>>>>>> I didn’t see some useful info.
>>>>>>
>>>>>> In mesos slave log, there is a line :
>>>>>> I1029 03:29:53.160143  9292 slave.cpp:3399] Executor
>>>>>> '279bcb34-f705-4857-96ad-d96843b848fb.4b3abdcd-7ded-11e5-a82d-0240afabf713'
>>>>>> of framework 20151029-031549-1294671788-5050-4937-0000 terminated
>>>>>> with signal Killed
>>>>>>
>>>>>> I check the normal log, it shows:
>>>>>>
>>>>>> I1014 15:22:21.276007 23163 slave.cpp:3326] Executor
>>>>>> 'ffc08dce-997f-41f7-9b03-57c1b4bc1f85.47ed02aa-7285-11e5-80d7-000d3a8033de'
>>>>>> of framework 20150814-115157-1677721866-5050-6185-0000 exited with
>>>>>> status 0
>>>>>>
>>>>>> Is this helpful?
>>>>>>
>>>>>> 发件人: Xiaodong Zhang <xdzhang@alauda.io>
>>>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>>> 日期: 2015年10月29日 星期四 下午12:59
>>>>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>>>
>>>>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>>>>
>>>>>> <9D46724C-457C-4BE1-B0E4-F57B147F6DC8.png>
>>>>>>
>>>>>> The webui have a LOG link, when click it shows like this:
>>>>>>
>>>>>> I1029 04:44:32.293445  5697 http.cpp:321] HTTP GET for
>>>>>> /master/state.json from 114.113.20.135:55682 with
>>>>>> User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5)
>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36'
>>>>>> I1029 04:44:34.533504  5704 master.cpp:4613] Sending 1 offers to
>>>>>> framework 20151029-043755-3549436724-5050-5674-0000 (marathon) at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> I1029 04:44:34.539579  5702 master.cpp:2739] Processing ACCEPT call
>>>>>> for offers: [ 20151029-043755-3549436724-5050-5674-O2 ] on slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 at slave(1)@
>>>>>> 50.112.136.148:5051 (
>>>>>> ec2-50-112-136-148.us-west-2.compute.amazonaws.com) for framework
>>>>>> 20151029-043755-3549436724-5050-5674-0000 (marathon) at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> I1029 04:44:34.539710  5702 hierarchical.hpp:814] Recovered
>>>>>> cpus(*):1; mem(*):999; disk(*):3962; ports(*):[31000-32000] (total:
>>>>>> cpus(*):1; mem(*):999; disk(*):3962; ports(*):[31000-32000], allocated:
)
>>>>>> on slave 20151029-043755-3549436724-5050-5674-S0 from framework
>>>>>> 20151029-043755-3549436724-5050-5674-0000
>>>>>> I1029 04:44:37.360901  5703 master.cpp:4294] Performing implicit
task
>>>>>> state reconciliation for framework
>>>>>> 20151029-043755-3549436724-5050-5674-0000 (marathon) at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> I1029 04:44:40.539989  5704 master.cpp:4613] Sending 1 offers to
>>>>>> framework 20151029-043755-3549436724-5050-5674-0000 (marathon) at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> I1029 04:44:40.610321  5702 master.cpp:2739] Processing ACCEPT call
>>>>>> for offers: [ 20151029-043755-3549436724-5050-5674-O3 ] on slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 at slave(1)@
>>>>>> 50.112.136.148:5051 (
>>>>>> ec2-50-112-136-148.us-west-2.compute.amazonaws.com) for framework
>>>>>> 20151029-043755-3549436724-5050-5674-0000 (marathon) at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> I1029 04:44:40.610846  5702 master.hpp:170] Adding task
>>>>>> e4a3bed5-64e6-4970-8bb1-df6404656a48.c4239b84-7df7-11e5-b57b-0247b493b22f
>>>>>> with resources cpus(*):0.0625; mem(*):256; ports(*):[31864-31864]
on slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 (
>>>>>> ec2-50-112-136-148.us-west-2.compute.amazonaws.com)
>>>>>> I1029 04:44:40.610911  5702 master.cpp:3069] Launching task
>>>>>> e4a3bed5-64e6-4970-8bb1-df6404656a48.c4239b84-7df7-11e5-b57b-0247b493b22f
>>>>>> of framework 20151029-043755-3549436724-5050-5674-0000 (marathon)
at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> with resources cpus(*):0.0625; mem(*):256; ports(*):[31864-31864]
on slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 at slave(1)@
>>>>>> 50.112.136.148:5051 (
>>>>>> ec2-50-112-136-148.us-west-2.compute.amazonaws.com)
>>>>>> I1029 04:44:40.611095  5702 hierarchical.hpp:814] Recovered
>>>>>> cpus(*):0.9375; mem(*):743; disk(*):3962; ports(*):[31000-31863,
>>>>>> 31865-32000] (total: cpus(*):1; mem(*):999; disk(*):3962;
>>>>>> ports(*):[31000-32000], allocated: cpus(*):0.0625; mem(*):256;
>>>>>> ports(*):[31864-31864]) on slave 20151029-043755-3549436724-5050-5674-S0
>>>>>> from framework 20151029-043755-3549436724-5050-5674-0000
>>>>>> I1029 04:44:43.324970  5698 http.cpp:321] HTTP GET for
>>>>>> /master/state.json from 114.113.20.135:55682 with
>>>>>> User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5)
>>>>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36'
>>>>>> I1029 04:44:46.546671  5703 master.cpp:4613] Sending 1 offers to
>>>>>> framework 20151029-043755-3549436724-5050-5674-0000 (marathon) at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> I1029 04:44:46.557266  5699 master.cpp:2739] Processing ACCEPT call
>>>>>> for offers: [ 20151029-043755-3549436724-5050-5674-O4 ] on slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 at slave(1)@
>>>>>> 50.112.136.148:5051 (
>>>>>> ec2-50-112-136-148.us-west-2.compute.amazonaws.com) for framework
>>>>>> 20151029-043755-3549436724-5050-5674-0000 (marathon) at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
>>>>>> I1029 04:44:46.557394  5699 hierarchical.hpp:814] Recovered
>>>>>> cpus(*):0.9375; mem(*):743; disk(*):3962; ports(*):[31000-31863,
>>>>>> 31865-32000] (total: cpus(*):1; mem(*):999; disk(*):3962;
>>>>>> ports(*):[31000-32000], allocated: cpus(*):0.0625; mem(*):256;
>>>>>> ports(*):[31864-31864]) on slave 20151029-043755-3549436724-5050-5674-S0
>>>>>> from framework 20151029-043755-3549436724-5050-5674-0000
>>>>>> I1029 04:44:47.267562  5700 master.cpp:4069] Status update
>>>>>> TASK_FAILED (UUID: 0ea607fc-bf24-4bda-b107-55a54aba31cf) for task
>>>>>> e4a3bed5-64e6-4970-8bb1-df6404656a48.c4239b84-7df7-11e5-b57b-0247b493b22f
>>>>>> of framework 20151029-043755-3549436724-5050-5674-0000 from slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 at slave(1)@
>>>>>> 50.112.136.148:5051 (
>>>>>> ec2-50-112-136-148.us-west-2.compute.amazonaws.com)
>>>>>> I1029 04:44:47.267645  5700 master.cpp:4108] Forwarding status update
>>>>>> TASK_FAILED (UUID: 0ea607fc-bf24-4bda-b107-55a54aba31cf) for task
>>>>>> e4a3bed5-64e6-4970-8bb1-df6404656a48.c4239b84-7df7-11e5-b57b-0247b493b22f
>>>>>> of framework 20151029-043755-3549436724-5050-5674-0000
>>>>>> I1029 04:44:47.267774  5700 master.cpp:5576] Updating the latest
>>>>>> state of task
>>>>>> e4a3bed5-64e6-4970-8bb1-df6404656a48.c4239b84-7df7-11e5-b57b-0247b493b22f
>>>>>> of framework 20151029-043755-3549436724-5050-5674-0000 to TASK_FAILED
>>>>>> I1029 04:44:47.267907  5700 hierarchical.hpp:814] Recovered
>>>>>> cpus(*):0.0625; mem(*):256; ports(*):[31864-31864] (total: cpus(*):1;
>>>>>> mem(*):999; disk(*):3962; ports(*):[31000-32000], allocated: ) on
slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 from framework
>>>>>> 20151029-043755-3549436724-5050-5674-0000
>>>>>> I1029 04:44:47.289356  5698 master.cpp:5644] Removing task
>>>>>> e4a3bed5-64e6-4970-8bb1-df6404656a48.c4239b84-7df7-11e5-b57b-0247b493b22f
>>>>>> with resources cpus(*):0.0625; mem(*):256; ports(*):[31864-31864]
of
>>>>>> framework 20151029-043755-3549436724-5050-5674-0000 on slave
>>>>>> 20151029-043755-3549436724-5050-5674-S0 at slave(1)@
>>>>>> 50.112.136.148:5051 (
>>>>>> ec2-50-112-136-148.us-west-2.compute.amazonaws.com)
>>>>>> I1029 04:44:47.289459  5698 master.cpp:3398] Processing ACKNOWLEDGE
>>>>>> call 0ea607fc-bf24-4bda-b107-55a54aba31cf for task
>>>>>> e4a3bed5-64e6-4970-8bb1-df6404656a48.c4239b84-7df7-11e5-b57b-0247b493b22f
>>>>>> of framework 20151029-043755-3549436724-5050-5674-0000 (marathon)
at
>>>>>> scheduler-b532233f-2fc5-4455-b1e6-7a66ae79a8b9@172.31.43.77:53373
on
>>>>>> slave 20151029-043755-3549436724-5050-5674-S0
>>>>>>
>>>>>>
>>>>>>
>>>>>> 发件人: haosdent <haosdent@gmail.com>
>>>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>>> 日期: 2015年10月29日 星期四 下午12:02
>>>>>> 至: user <user@mesos.apache.org>
>>>>>> 主题: Re: Can't start docker container when SSL_ENABLED is on.
>>>>>>
>>>>>> Oh, I mean you task logs. They could be get from Mesos webui.
>>>>>>
>>>>>> On Thu, Oct 29, 2015 at 11:52 AM, Xiaodong Zhang <xdzhang@alauda.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for your reply.
>>>>>>>
>>>>>>> Yes I build mesos with `--enable-libevent --enable-ssl`. If I
don’t
>>>>>>> provide key and pem when start slave, it will register fail(That
means the
>>>>>>> ssl work well right?)
>>>>>>>
>>>>>>> As I said the odd thing is the container nerver run(`docker ps
–a
>>>>>>> show nothing`). So it can’t have any stdout or stderr.
>>>>>>>
>>>>>>> 发件人: haosdent <haosdent@gmail.com>
>>>>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>>>> 日期: 2015年10月29日 星期四 上午11:47
>>>>>>> 至: user <user@mesos.apache.org>
>>>>>>> 主题: Re: Can't start docker container when SSL_ENABLED is
on.
>>>>>>>
>>>>>>> Do you compile mesos with ssl support? The default compile don't
>>>>>>> contains ssl. And does docker container have stdour and stderr?
>>>>>>>
>>>>>>> On Thu, Oct 29, 2015 at 11:41 AM, Xiaodong Zhang <xdzhang@alauda.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> My scenarios is like previous email says, masters and slaves
are in
>>>>>>>> different IaaS. Now the slaves can register to the masters
with SSL_ENABLED
>>>>>>>> is on .
>>>>>>>>
>>>>>>>> But I meet another problem. Slaves can’t run container(the
odd
>>>>>>>> thing is they can pull image successfully,just can not run
container,
>>>>>>>> `docker ps –a ` list nothing)
>>>>>>>>
>>>>>>>> The logs like this:
>>>>>>>>
>>>>>>>> I1029 03:29:45.967741  9288 docker.cpp:758] Starting container
>>>>>>>> 'd4f4e236-0d0a-492c-86df-eef48a414e23' for task
>>>>>>>> '279bcb34-f705-4857-96ad-d96843b848fb.4b3abdcd-7ded-11e5-a82d-0240afabf713'
>>>>>>>> (and executor
>>>>>>>> '279bcb34-f705-4857-96ad-d96843b848fb.4b3abdcd-7ded-11e5-a82d-0240afabf713')
>>>>>>>> of framework '20151029-031549-1294671788-5050-4937-0000'
>>>>>>>> I1029 03:29:48.044148  9292 docker.cpp:382] Checkpointing
pid 12062
>>>>>>>> to
>>>>>>>> '/tmp/mesos/meta/slaves/20151029-031549-1294671788-5050-4937-S0/frameworks/20151029-031549-1294671788-5050-4937-0000/executors/279bcb34-f705-4857-96ad-d96843b848fb.4b3abdcd-7ded-11e5-a82d-0240afabf713/runs/d4f4e236-0d0a-492c-86df-eef48a414e23/pids/forked.pid'
>>>>>>>> I1029 03:29:53.159361  9292 docker.cpp:1576] Executor for
container
>>>>>>>> 'd4f4e236-0d0a-492c-86df-eef48a414e23' has exited
>>>>>>>> I1029 03:29:53.159572  9292 docker.cpp:1374] Destroying container
>>>>>>>> 'd4f4e236-0d0a-492c-86df-eef48a414e23'
>>>>>>>> I1029 03:29:53.159822  9292 docker.cpp:1478] Running docker
stop on
>>>>>>>> container 'd4f4e236-0d0a-492c-86df-eef48a414e23'
>>>>>>>> I1029 03:29:53.160143  9292 slave.cpp:3399] Executor
>>>>>>>> '279bcb34-f705-4857-96ad-d96843b848fb.4b3abdcd-7ded-11e5-a82d-0240afabf713'
>>>>>>>> of framework 20151029-031549-1294671788-5050-4937-0000 terminated
>>>>>>>> with signal Killed
>>>>>>>> I1029 03:29:53.160884  9292 slave.cpp:2696] Handling status
update
>>>>>>>> TASK_FAILED (UUID: 27a2080a-8807-449e-9077-837ec45b4c51)
for task
>>>>>>>> 279bcb34-f705-4857-96ad-d96843b848fb.4b3abdcd-7ded-11e5-a82d-0240afabf713
>>>>>>>> of framework 20151029-031549-1294671788-5050-4937-0000 from
@
>>>>>>>> 0.0.0.0:0
>>>>>>>> W1029 03:29:53.161247  9288 docker.cpp:986] Ignoring updating
>>>>>>>> unknown container: d4f4e236-0d0a-492c-86df-eef48a414e23
>>>>>>>> I1029 03:29:53.161548  9293 status_update_manager.cpp:322]
Received
>>>>>>>> status update TASK_FAILED (UUID: 27a2080a-8807-449e-9077-837ec45b4c51)
for
>>>>>>>> task
>>>>>>>> 279bcb34-f705-4857-96ad-d96843b848fb.4b3abdcd-7ded-11e5-a82d-0240afabf713
>>>>>>>> of framework 20151029-031549-1294671788-5050-4937-0000
>>>>>>>>
>>>>>>>> I run master node with env:
>>>>>>>>
>>>>>>>> SSL_SUPPORT_DOWNGRADE=true
>>>>>>>> SSL_ENABLED=true
>>>>>>>> SSL_KEY_FILE=/home/ubuntu/xx.key
>>>>>>>> SSL_CERT_FILE=/home/ubuntu/xx.pem
>>>>>>>>
>>>>>>>> Slave node with env:
>>>>>>>>
>>>>>>>> SSL_ENABLED=true
>>>>>>>> SSL_KEY_FILE=/home/ubuntu/xx.key
>>>>>>>> SSL_CERT_FILE=/home/ubuntu/xx.pem
>>>>>>>> LIBPROCESS_ADVERTISE_IP=xxx.xxx.xxx.xxx
>>>>>>>>
>>>>>>>> When I remove all SSL envs. Slaves work well.
>>>>>>>>
>>>>>>>> Did I miss sth?
>>>>>>>>
>>>>>>>> Version:
>>>>>>>>
>>>>>>>> Mesos 0.24.1
>>>>>>>> Maraton 0.9.2
>>>>>>>>
>>>>>>>> OS
>>>>>>>> ubuntu 14.04
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 发件人: Anindya Sinha <anindya.sinha@gmail.com>
>>>>>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>>>>> 日期: 2015年10月28日 星期三 下午2:32
>>>>>>>> 至: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>>>>> 主题: Re: How to tell master which ip to connect.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 27, 2015 at 7:43 PM, Xiaodong Zhang <xdzhang@alauda.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> It works! Thanks a lot.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Ok. So we should expose advertise_ip and advertise_port as
command
>>>>>>>> line options for mesos-slave as well (instead of using the
environment
>>>>>>>> variables)? Opened https://issues.apache.org/jira/browse/MESOS-3809
>>>>>>>> .
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Another question. Do masters and slaves communicate each
other via
>>>>>>>>> a safety way?Is the data encrypted? I want to make
sure deploy masters and
>>>>>>>>> slaves into different IaaS is PROD-READY.
>>>>>>>>>
>>>>>>>>> 发件人: haosdent <haosdent@gmail.com>
>>>>>>>>> 答复: "user@mesos.apache.org" <user@mesos.apache.org>
>>>>>>>>> 日期: 2015年10月28日 星期三 上午10:23
>>>>>>>>> 至: user <user@mesos.apache.org>
>>>>>>>>> 主题: Re: How to tell master which ip to connect.
>>>>>>>>>
>>>>>>>>> Do you try `export LIBPROCESS_ADVERTISE_IP=xxx` and
>>>>>>>>> `LIBPROCESS_ADVERTISE_PORT` when start slave?
>>>>>>>>>
>>>>>>>>> On Wed, Oct 28, 2015 at 10:16 AM, Xiaodong Zhang <
>>>>>>>>> xdzhang@alauda.io> wrote:
>>>>>>>>>
>>>>>>>>>> Hi teams:
>>>>>>>>>>
>>>>>>>>>> My scenarios is like this:
>>>>>>>>>>
>>>>>>>>>> My master nodes were deployed in AWS. My slaves were
in AZURE.So
>>>>>>>>>> they communicate via public ip.
>>>>>>>>>> I got trouble when slaves try to register to master.
>>>>>>>>>> Now slaves can get master’s public ip address,and
can send
>>>>>>>>>> register request.But they can only send there private
ip to master.(Because
>>>>>>>>>> they don’t know there public ip,thus they can’t
not bind a public ip via
>>>>>>>>>> —ip flag), thus  masters can’t connect slaves.How
can the slave to tell
>>>>>>>>>> master which ip master should connect(I can’t find
any flags like —advertise_ip
>>>>>>>>>> in master).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>> Haosdent Huang
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Haosdent Huang
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Haosdent Huang
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Haosdent Huang
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>> <5185_02_07.png><9D46724C-457C-4BE1-B0E4-F57B147F6DC8.png>
>>> <742629F2-78E8-43F2-9015-F3D22720826B.png><5185_02_04.png>
>>>
>>>
>>>
>>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>



-- 
Best Regards,
Haosdent Huang

Mime
View raw message