mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From haosdent <haosd...@gmail.com>
Subject Re: Can health-checks be run by Mesos for docker tasks?
Date Wed, 07 Oct 2015 06:24:36 GMT
Maybe you could post your executor stdout/stderr so that we could know
whether health check running not.

On Wed, Oct 7, 2015 at 2:15 PM, haosdent <haosdent@gmail.com> wrote:

> marathon also use mesos health check. When I use health check, I could saw
> the log like this in executor stdout.
>
> ```
> Registered docker executor on xxxxx
> Starting task test-health-check.822a5fd2-6cba-11e5-b5ce-0a0027000000
> Launching health check process:
> /home/haosdent/mesos/build/src/.libs/mesos-health-check --executor=xxxx
> Health check process launched at pid: 9895
> Received task health update, healthy: true
> ```
>
> On Wed, Oct 7, 2015 at 12:51 PM, Jay Taylor <outtatime@gmail.com> wrote:
>
>> I am using my own framework, and the full task info I'm using is posted
>> earlier in this thread.  Do you happen to know if Marathon uses Mesos's
>> health checks for its health check system?
>>
>>
>>
>> On Oct 6, 2015, at 9:01 PM, haosdent <haosdent@gmail.com> wrote:
>>
>> Yes, launch the health task through its definition in taskinfo. Do you
>> launch your task through Marathon? I could test it in my side.
>>
>> On Wed, Oct 7, 2015 at 11:56 AM, Jay Taylor <outtatime@gmail.com> wrote:
>>
>>> Precisely, and there are none of those statements.  Are you or others
>>> confident health-checks are part of the code path when defined via task
>>> info for docker container tasks?  Going through the code, I wasn't able to
>>> find the linkage for anything other than health-checks triggered through a
>>> custom executor.
>>>
>>> With that being said it is a pretty good sized code base and I'm not
>>> very familiar with it, so my analysis this far has by no means been
>>> exhaustive.
>>>
>>>
>>>
>>> On Oct 6, 2015, at 8:41 PM, haosdent <haosdent@gmail.com> wrote:
>>>
>>> When health check launch, it would have a log like this in your executor
>>> stdout
>>> ```
>>> Health check process launched at pid xxx
>>> ```
>>>
>>> On Wed, Oct 7, 2015 at 11:37 AM, Jay Taylor <outtatime@gmail.com> wrote:
>>>
>>>> I'm happy to try this, however wouldn't there be output in the logs
>>>> with the string "health" or "Health" if the health-check were active?  None
>>>> of my master or slave logs contain the string..
>>>>
>>>>
>>>>
>>>> On Oct 6, 2015, at 7:45 PM, haosdent <haosdent@gmail.com> wrote:
>>>>
>>>> Could you use "exit 1" instead of "sleep 5" to see whether could see
>>>> unhealthy status in your task stdout/stderr.
>>>>
>>>> On Wed, Oct 7, 2015 at 10:38 AM, Jay Taylor <outtatime@gmail.com>
>>>> wrote:
>>>>
>>>>> My current version is 0.24.1.
>>>>>
>>>>> On Tue, Oct 6, 2015 at 7:30 PM, haosdent <haosdent@gmail.com> wrote:
>>>>>
>>>>>> yes, adam also help commit it to 0.23.1 and 0.24.1
>>>>>> https://github.com/apache/mesos/commit/8c0ed92de3925d4312429bfba01b9b1ccbcbbef0
>>>>>>
>>>>>> https://github.com/apache/mesos/commit/09e367cd69aa39c156c9326d44f4a7b829ba3db7
>>>>>> Are you use one of this version?
>>>>>>
>>>>>> On Wed, Oct 7, 2015 at 10:26 AM, haosdent <haosdent@gmail.com>
wrote:
>>>>>>
>>>>>>> I remember 0.23.1 and 0.24.1 contains this backport, let me double
>>>>>>> check.
>>>>>>>
>>>>>>> On Wed, Oct 7, 2015 at 10:01 AM, Jay Taylor <outtatime@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Oops- Now I see you already said it's in master.  I'll look
there :)
>>>>>>>>
>>>>>>>> Thanks again!
>>>>>>>>
>>>>>>>> On Tue, Oct 6, 2015 at 6:59 PM, Jay Taylor <jay@jaytaylor.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Great, thanks for the quick reply Tim!
>>>>>>>>>
>>>>>>>>> Do you know if there is a branch I can checkout to test
it out?
>>>>>>>>>
>>>>>>>>> On Tue, Oct 6, 2015 at 6:54 PM, Timothy Chen <tim@mesosphere.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Jay,
>>>>>>>>>>
>>>>>>>>>> We just added health check support for docker tasks
that's in
>>>>>>>>>> master but not yet released. It will run docker exec
with the command you
>>>>>>>>>> provided as health checks.
>>>>>>>>>>
>>>>>>>>>> It should be in the next release.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Tim
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Oct 6, 2015, at 6:49 PM, Jay Taylor <outtatime@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Does Mesos support health checks for docker image
tasks?  Mesos
>>>>>>>>>> seems to be ignoring the TaskInfo.HealthCheck field
for me.
>>>>>>>>>>
>>>>>>>>>> Example TaskInfo JSON received back from Mesos:
>>>>>>>>>>
>>>>>>>>>> {
>>>>>>>>>>>
>>>>>>>>>>>   "name":"hello-app.web.v3",
>>>>>>>>>>>
>>>>>>>>>>>   "task_id":{
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> "value":"hello-app_web-v3.fc05a1a5-1e06-4e61-9879-be0d97cd3eec"
>>>>>>>>>>>
>>>>>>>>>>>   },
>>>>>>>>>>>
>>>>>>>>>>>   "slave_id":{
>>>>>>>>>>>
>>>>>>>>>>>     "value":"20150924-210922-1608624320-5050-1792-S1"
>>>>>>>>>>>
>>>>>>>>>>>   },
>>>>>>>>>>>
>>>>>>>>>>>   "resources":[
>>>>>>>>>>>
>>>>>>>>>>>     {
>>>>>>>>>>>
>>>>>>>>>>>       "name":"cpus",
>>>>>>>>>>>
>>>>>>>>>>>       "type":0,
>>>>>>>>>>>
>>>>>>>>>>>       "scalar":{
>>>>>>>>>>>
>>>>>>>>>>>         "value":0.1
>>>>>>>>>>>
>>>>>>>>>>>       }
>>>>>>>>>>>
>>>>>>>>>>>     },
>>>>>>>>>>>
>>>>>>>>>>>     {
>>>>>>>>>>>
>>>>>>>>>>>       "name":"mem",
>>>>>>>>>>>
>>>>>>>>>>>       "type":0,
>>>>>>>>>>>
>>>>>>>>>>>       "scalar":{
>>>>>>>>>>>
>>>>>>>>>>>         "value":256
>>>>>>>>>>>
>>>>>>>>>>>       }
>>>>>>>>>>>
>>>>>>>>>>>     },
>>>>>>>>>>>
>>>>>>>>>>>     {
>>>>>>>>>>>
>>>>>>>>>>>       "name":"ports",
>>>>>>>>>>>
>>>>>>>>>>>       "type":1,
>>>>>>>>>>>
>>>>>>>>>>>       "ranges":{
>>>>>>>>>>>
>>>>>>>>>>>         "range":[
>>>>>>>>>>>
>>>>>>>>>>>           {
>>>>>>>>>>>
>>>>>>>>>>>             "begin":31002,
>>>>>>>>>>>
>>>>>>>>>>>             "end":31002
>>>>>>>>>>>
>>>>>>>>>>>           }
>>>>>>>>>>>
>>>>>>>>>>>         ]
>>>>>>>>>>>
>>>>>>>>>>>       }
>>>>>>>>>>>
>>>>>>>>>>>     }
>>>>>>>>>>>
>>>>>>>>>>>   ],
>>>>>>>>>>>
>>>>>>>>>>>   "command":{
>>>>>>>>>>>
>>>>>>>>>>>     "container":{
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> "image":"docker-services1a:5000/test/app-81-1-hello-app-103"
>>>>>>>>>>>
>>>>>>>>>>>     },
>>>>>>>>>>>
>>>>>>>>>>>     "shell":false
>>>>>>>>>>>
>>>>>>>>>>>   },
>>>>>>>>>>>
>>>>>>>>>>>   "container":{
>>>>>>>>>>>
>>>>>>>>>>>     "type":1,
>>>>>>>>>>>
>>>>>>>>>>>     "docker":{
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> "image":"docker-services1a:5000/gig1/app-81-1-hello-app-103",
>>>>>>>>>>>
>>>>>>>>>>>       "network":2,
>>>>>>>>>>>
>>>>>>>>>>>       "port_mappings":[
>>>>>>>>>>>
>>>>>>>>>>>         {
>>>>>>>>>>>
>>>>>>>>>>>           "host_port":31002,
>>>>>>>>>>>
>>>>>>>>>>>           "container_port":8000,
>>>>>>>>>>>
>>>>>>>>>>>           "protocol":"tcp"
>>>>>>>>>>>
>>>>>>>>>>>         }
>>>>>>>>>>>
>>>>>>>>>>>       ],
>>>>>>>>>>>
>>>>>>>>>>>       "privileged":false,
>>>>>>>>>>>
>>>>>>>>>>>       "parameters":[],
>>>>>>>>>>>
>>>>>>>>>>>       "force_pull_image":false
>>>>>>>>>>>
>>>>>>>>>>>     }
>>>>>>>>>>>
>>>>>>>>>>>   },
>>>>>>>>>>>
>>>>>>>>>>>   "health_check":{
>>>>>>>>>>>
>>>>>>>>>>>     "delay_seconds":5,
>>>>>>>>>>>
>>>>>>>>>>>     "interval_seconds":10,
>>>>>>>>>>>
>>>>>>>>>>>     "timeout_seconds":10,
>>>>>>>>>>>
>>>>>>>>>>>     "consecutive_failures":3,
>>>>>>>>>>>
>>>>>>>>>>>     "grace_period_seconds":0,
>>>>>>>>>>>
>>>>>>>>>>>     "command":{
>>>>>>>>>>>
>>>>>>>>>>>       "shell":true,
>>>>>>>>>>>
>>>>>>>>>>>       "value":"sleep 5",
>>>>>>>>>>>
>>>>>>>>>>>       "user":"root"
>>>>>>>>>>>
>>>>>>>>>>>     }
>>>>>>>>>>>
>>>>>>>>>>>   }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I have searched all machines and containers to see
if they ever
>>>>>>>>>> run the command (in this case `sleep 5`), but have
not found any indication
>>>>>>>>>> that it is being executed.
>>>>>>>>>>
>>>>>>>>>> In the mesos src code the health-checks are invoked
from
>>>>>>>>>> src/launcher/executor.cpp CommandExecutorProcess::launchTask.
 Does this
>>>>>>>>>> mean that health-checks are only supported for custom
executors and not for
>>>>>>>>>> docker tasks?
>>>>>>>>>>
>>>>>>>>>> What I am trying to accomplish is to have the 0/non-zero
>>>>>>>>>> exit-status of a health-check command translate to
task health.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> Jay
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Haosdent Huang
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards,
>>>>>> Haosdent Huang
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Haosdent Huang
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>>
>
>
> --
> Best Regards,
> Haosdent Huang
>



-- 
Best Regards,
Haosdent Huang

Mime
View raw message