mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From haosdent <haosd...@gmail.com>
Subject Re: Completed executors presented as alive
Date Sat, 04 Jun 2016 03:35:09 GMT
> 13:33:39.031054  [slave.cpp:2643] Got registration for executor
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 from executor(1)@
10.55.97.170:60083

Yes, according to your log, your executor is still running. If your
executor is http_command_executor,
you could use
https://github.com/apache/mesos/blob/master/docs/executor-http-api.md#shutdown
to shutdown it.
If it is other type executor, seems don't have a api to shutdown executor
as I know. Not sure whether kill the executor in
Agent could resolve your problem or not.

On Fri, Jun 3, 2016 at 4:33 PM, Tomek Janiszewski <janiszt@gmail.com> wrote:

> Here is truncated response from slave(1)/state
>
> {
>     "attributes": {...},
>     "completed_frameworks": [],
>     "flags": {...},
>     "frameworks": [
>         {
>             "checkpoint": true,
>             "completed_executors": [...],
>             "executors": [
>               {
>                   "queued_tasks": [],
>                   "tasks": [],
>                   "completed_tasks": [
>                       {
>                           "discovery": {...},
>                           "executor_id": "",
>                           "framework_id":
> "f65b163c-0faf-441f-ac14-91739fa4394c-0000",
>                           "id":
> "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
>                           "labels": [...],
>                           "name": "service",
>                           "resources": {...},
>                           "slave_id":
> "ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13",
>                           "state": "TASK_KILLED",
>                           "statuses": []
>                       }
>                   ],
>                   "container": "ead42e63-ac92-4ad0-a99c-4af9c3fa5e31",
>                   "directory": "...",
>                   "id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
>                   "name": "Command Executor (Task:
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e) (Command: sh -c 'cd
> service...')",
>                   "resources": {...},
>                   "source": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e"
>
>               },
>               ...
>             ],
>         }
>     ],
>     "git_sha": "961edbd82e691a619a4c171a7aadc9c32957fa73",
>     "git_tag": "0.28.0",
>     "version": "0.28.0",
>     ...
> }
>
> Here is the log for this container:
>
> > 13:33:19.479182  [slave.cpp:1361] Got assigned task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> > 13:33:19.482566  [slave.cpp:1480] Launching task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> > 13:33:19.483921  [paths.cpp:528] Trying to chown
>
> '/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
> to user 'mesosuser'
> > 13:33:19.504173  [slave.cpp:5367] Launching executor
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 with resources cpus(*):0.1;
> mem(*):32 in work directory
>
> '/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
> > 13:33:19.505537  [containerizer.cpp:666] Starting container
> 'ead42e63-ac92-4ad0-a99c-4af9c3fa5e31' for executor
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
> 'f65b163c-0faf-441f-ac14-91739fa4394c-0000'
> > 13:33:19.505734  [slave.cpp:1698] Queuing task
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' for executor
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> ...
> > 13:33:19.977483  [containerizer.cpp:1118] Checkpointing executor's forked
> pid 25576 to
>
> '/tmp/mesos/meta/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31/pids/forked.pid'
> > 13:33:35.775195  [slave.cpp:1891] Asked to kill task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> > 13:33:35.775645  [slave.cpp:3002] Handling status update TASK_KILLED
> (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 f
> rom @0.0.0.0:0
> > 13:33:35.778105  [cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus
> 0.1) for container ead42e63-ac92-4ad0-a99c-4af9c3fa5e31
> > 13:33:35.778488  [disk.cpp:169] Updating the disk resources for container
> ead42e63-ac92-4ad0-a99c-4af9c3fa5e31 to cpus(*):0.1
> ; mem(*):32
> > 13:33:35.780349  [mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to
> 32MB for container ead42e63-ac92-4ad0-a99c-4af9c3fa5e3
> 1
> > 13:33:35.782573  [status_update_manager.cpp:320] Received status update
> TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a8
> 1b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-9173
> 9fa4394c-0000
> > 13:33:35.783860  [status_update_manager.cpp:824] Checkpointing UPDATE for
> status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for
> task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> > 13:33:35.788767  [slave.cpp:3400] Forwarding the update TASK_KILLED
> (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 to master@10.82.24.138:5050
> > 13:33:35.917932  [status_update_manager.cpp:392] Received status update
> acknowledgement (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> > 13:33:35.918143  [status_update_manager.cpp:824] Checkpointing ACK for
> status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for
> task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> ...
> > 13:33:39.031054  [slave.cpp:2643] Got registration for executor
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 from executor(1)@
> 10.55.97.170:60083
>
>
> Visible container is no longer running but it appears as running. What
> should I do with it?
>
> Thanks
> Tomek
>
>
> czw., 2.06.2016 o 15:55 użytkownik Tomek Janiszewski <janiszt@gmail.com>
> napisał:
>
> > Yes. I see dead executor in executors. It's tasks and queued_tasks are
> > empty but there is one task in completed_tasks.
> frameworks.completed_executors
> > are filled with other executors.
> >
> > czw., 2.06.2016 o 15:39 użytkownik haosdent <haosdent@gmail.com>
> napisał:
> >
> >> Hi, @janiszt Seems the completed executors only exists
> >> in completed_frameworks.completed_executors
> >> or frameworks.completed_executors in my side.
> >>
> >> In your side, does completed_executors exists in any other fields?
> >>
> >> On Thu, Jun 2, 2016 at 5:39 PM, Tomek Janiszewski <janiszt@gmail.com>
> >> wrote:
> >>
> >> > Hi
> >> >
> >> > I'm running Mesos 0.28.0. Mesos slave(1)/state endpoint returns some
> >> > completed executors not in frameworks.completed_executors but in
> >> > frameworks.
> >> > executors.
> >> > Is it normal behavior? How to force Mesos to move completed
> >> > executors into frameworks.executors?
> >> >
> >> > Thanks
> >> > Tomek
> >> >
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Haosdent Huang
> >>
> >
>



-- 
Best Regards,
Haosdent Huang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message