mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From haosdent <haosd...@gmail.com>
Subject Re: Can health-checks be run by Mesos for docker tasks?
Date Wed, 07 Oct 2015 06:15:20 GMT
marathon also use mesos health check. When I use health check, I could saw
the log like this in executor stdout.

```
Registered docker executor on xxxxx
Starting task test-health-check.822a5fd2-6cba-11e5-b5ce-0a0027000000
Launching health check process:
/home/haosdent/mesos/build/src/.libs/mesos-health-check --executor=xxxx
Health check process launched at pid: 9895
Received task health update, healthy: true
```

On Wed, Oct 7, 2015 at 12:51 PM, Jay Taylor <outtatime@gmail.com> wrote:

> I am using my own framework, and the full task info I'm using is posted
> earlier in this thread.  Do you happen to know if Marathon uses Mesos's
> health checks for its health check system?
>
>
>
> On Oct 6, 2015, at 9:01 PM, haosdent <haosdent@gmail.com> wrote:
>
> Yes, launch the health task through its definition in taskinfo. Do you
> launch your task through Marathon? I could test it in my side.
>
> On Wed, Oct 7, 2015 at 11:56 AM, Jay Taylor <outtatime@gmail.com> wrote:
>
>> Precisely, and there are none of those statements.  Are you or others
>> confident health-checks are part of the code path when defined via task
>> info for docker container tasks?  Going through the code, I wasn't able to
>> find the linkage for anything other than health-checks triggered through a
>> custom executor.
>>
>> With that being said it is a pretty good sized code base and I'm not very
>> familiar with it, so my analysis this far has by no means been exhaustive.
>>
>>
>>
>> On Oct 6, 2015, at 8:41 PM, haosdent <haosdent@gmail.com> wrote:
>>
>> When health check launch, it would have a log like this in your executor
>> stdout
>> ```
>> Health check process launched at pid xxx
>> ```
>>
>> On Wed, Oct 7, 2015 at 11:37 AM, Jay Taylor <outtatime@gmail.com> wrote:
>>
>>> I'm happy to try this, however wouldn't there be output in the logs with
>>> the string "health" or "Health" if the health-check were active?  None of
>>> my master or slave logs contain the string..
>>>
>>>
>>>
>>> On Oct 6, 2015, at 7:45 PM, haosdent <haosdent@gmail.com> wrote:
>>>
>>> Could you use "exit 1" instead of "sleep 5" to see whether could see
>>> unhealthy status in your task stdout/stderr.
>>>
>>> On Wed, Oct 7, 2015 at 10:38 AM, Jay Taylor <outtatime@gmail.com> wrote:
>>>
>>>> My current version is 0.24.1.
>>>>
>>>> On Tue, Oct 6, 2015 at 7:30 PM, haosdent <haosdent@gmail.com> wrote:
>>>>
>>>>> yes, adam also help commit it to 0.23.1 and 0.24.1
>>>>> https://github.com/apache/mesos/commit/8c0ed92de3925d4312429bfba01b9b1ccbcbbef0
>>>>>
>>>>> https://github.com/apache/mesos/commit/09e367cd69aa39c156c9326d44f4a7b829ba3db7
>>>>> Are you use one of this version?
>>>>>
>>>>> On Wed, Oct 7, 2015 at 10:26 AM, haosdent <haosdent@gmail.com>
wrote:
>>>>>
>>>>>> I remember 0.23.1 and 0.24.1 contains this backport, let me double
>>>>>> check.
>>>>>>
>>>>>> On Wed, Oct 7, 2015 at 10:01 AM, Jay Taylor <outtatime@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Oops- Now I see you already said it's in master.  I'll look there
:)
>>>>>>>
>>>>>>> Thanks again!
>>>>>>>
>>>>>>> On Tue, Oct 6, 2015 at 6:59 PM, Jay Taylor <jay@jaytaylor.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Great, thanks for the quick reply Tim!
>>>>>>>>
>>>>>>>> Do you know if there is a branch I can checkout to test it
out?
>>>>>>>>
>>>>>>>> On Tue, Oct 6, 2015 at 6:54 PM, Timothy Chen <tim@mesosphere.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Jay,
>>>>>>>>>
>>>>>>>>> We just added health check support for docker tasks that's
in
>>>>>>>>> master but not yet released. It will run docker exec
with the command you
>>>>>>>>> provided as health checks.
>>>>>>>>>
>>>>>>>>> It should be in the next release.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Oct 6, 2015, at 6:49 PM, Jay Taylor <outtatime@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Does Mesos support health checks for docker image tasks?
 Mesos
>>>>>>>>> seems to be ignoring the TaskInfo.HealthCheck field for
me.
>>>>>>>>>
>>>>>>>>> Example TaskInfo JSON received back from Mesos:
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>>>
>>>>>>>>>>   "name":"hello-app.web.v3",
>>>>>>>>>>
>>>>>>>>>>   "task_id":{
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> "value":"hello-app_web-v3.fc05a1a5-1e06-4e61-9879-be0d97cd3eec"
>>>>>>>>>>
>>>>>>>>>>   },
>>>>>>>>>>
>>>>>>>>>>   "slave_id":{
>>>>>>>>>>
>>>>>>>>>>     "value":"20150924-210922-1608624320-5050-1792-S1"
>>>>>>>>>>
>>>>>>>>>>   },
>>>>>>>>>>
>>>>>>>>>>   "resources":[
>>>>>>>>>>
>>>>>>>>>>     {
>>>>>>>>>>
>>>>>>>>>>       "name":"cpus",
>>>>>>>>>>
>>>>>>>>>>       "type":0,
>>>>>>>>>>
>>>>>>>>>>       "scalar":{
>>>>>>>>>>
>>>>>>>>>>         "value":0.1
>>>>>>>>>>
>>>>>>>>>>       }
>>>>>>>>>>
>>>>>>>>>>     },
>>>>>>>>>>
>>>>>>>>>>     {
>>>>>>>>>>
>>>>>>>>>>       "name":"mem",
>>>>>>>>>>
>>>>>>>>>>       "type":0,
>>>>>>>>>>
>>>>>>>>>>       "scalar":{
>>>>>>>>>>
>>>>>>>>>>         "value":256
>>>>>>>>>>
>>>>>>>>>>       }
>>>>>>>>>>
>>>>>>>>>>     },
>>>>>>>>>>
>>>>>>>>>>     {
>>>>>>>>>>
>>>>>>>>>>       "name":"ports",
>>>>>>>>>>
>>>>>>>>>>       "type":1,
>>>>>>>>>>
>>>>>>>>>>       "ranges":{
>>>>>>>>>>
>>>>>>>>>>         "range":[
>>>>>>>>>>
>>>>>>>>>>           {
>>>>>>>>>>
>>>>>>>>>>             "begin":31002,
>>>>>>>>>>
>>>>>>>>>>             "end":31002
>>>>>>>>>>
>>>>>>>>>>           }
>>>>>>>>>>
>>>>>>>>>>         ]
>>>>>>>>>>
>>>>>>>>>>       }
>>>>>>>>>>
>>>>>>>>>>     }
>>>>>>>>>>
>>>>>>>>>>   ],
>>>>>>>>>>
>>>>>>>>>>   "command":{
>>>>>>>>>>
>>>>>>>>>>     "container":{
>>>>>>>>>>
>>>>>>>>>>       "image":"docker-services1a:5000/test/app-81-1-hello-app-103"
>>>>>>>>>>
>>>>>>>>>>     },
>>>>>>>>>>
>>>>>>>>>>     "shell":false
>>>>>>>>>>
>>>>>>>>>>   },
>>>>>>>>>>
>>>>>>>>>>   "container":{
>>>>>>>>>>
>>>>>>>>>>     "type":1,
>>>>>>>>>>
>>>>>>>>>>     "docker":{
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> "image":"docker-services1a:5000/gig1/app-81-1-hello-app-103",
>>>>>>>>>>
>>>>>>>>>>       "network":2,
>>>>>>>>>>
>>>>>>>>>>       "port_mappings":[
>>>>>>>>>>
>>>>>>>>>>         {
>>>>>>>>>>
>>>>>>>>>>           "host_port":31002,
>>>>>>>>>>
>>>>>>>>>>           "container_port":8000,
>>>>>>>>>>
>>>>>>>>>>           "protocol":"tcp"
>>>>>>>>>>
>>>>>>>>>>         }
>>>>>>>>>>
>>>>>>>>>>       ],
>>>>>>>>>>
>>>>>>>>>>       "privileged":false,
>>>>>>>>>>
>>>>>>>>>>       "parameters":[],
>>>>>>>>>>
>>>>>>>>>>       "force_pull_image":false
>>>>>>>>>>
>>>>>>>>>>     }
>>>>>>>>>>
>>>>>>>>>>   },
>>>>>>>>>>
>>>>>>>>>>   "health_check":{
>>>>>>>>>>
>>>>>>>>>>     "delay_seconds":5,
>>>>>>>>>>
>>>>>>>>>>     "interval_seconds":10,
>>>>>>>>>>
>>>>>>>>>>     "timeout_seconds":10,
>>>>>>>>>>
>>>>>>>>>>     "consecutive_failures":3,
>>>>>>>>>>
>>>>>>>>>>     "grace_period_seconds":0,
>>>>>>>>>>
>>>>>>>>>>     "command":{
>>>>>>>>>>
>>>>>>>>>>       "shell":true,
>>>>>>>>>>
>>>>>>>>>>       "value":"sleep 5",
>>>>>>>>>>
>>>>>>>>>>       "user":"root"
>>>>>>>>>>
>>>>>>>>>>     }
>>>>>>>>>>
>>>>>>>>>>   }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I have searched all machines and containers to see if
they ever
>>>>>>>>> run the command (in this case `sleep 5`), but have not
found any indication
>>>>>>>>> that it is being executed.
>>>>>>>>>
>>>>>>>>> In the mesos src code the health-checks are invoked from
>>>>>>>>> src/launcher/executor.cpp CommandExecutorProcess::launchTask.
 Does this
>>>>>>>>> mean that health-checks are only supported for custom
executors and not for
>>>>>>>>> docker tasks?
>>>>>>>>>
>>>>>>>>> What I am trying to accomplish is to have the 0/non-zero
>>>>>>>>> exit-status of a health-check command translate to task
health.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> Jay
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Haosdent Huang
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards,
>>>>> Haosdent Huang
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>
>


-- 
Best Regards,
Haosdent Huang

Mime
View raw message