incubator-mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Mahler (JIRA)" <>
Subject [jira] [Commented] (MESOS-433) add exportStats method to executor
Date Wed, 17 Apr 2013 17:21:16 GMT


Benjamin Mahler commented on MESOS-433:

This is starting to suggest that mesos is providing at least part of an observability stack
on behalf of executors, something that I'm not convinced is a good idea, merely in terms of
keeping down the number of features we provide. Presumably, each executor can host an endpoint
and the framework will know about all of the executors' endpoints. Whatever observability
stack is used can then ask each endpoint for the relevant data.

Of course, there are likely many frameworks that will need to implement this, so it may very
well be something we should implement, but I want to make sure the benefits are clear before
adding any complexity:
  1. One endpoint per slave, rather than multiple endpoints for the executors.
  2. Mesos will store the data on behalf of executors, but executors probably need to cache
this data anyway.
  3. ?

  1. Adding complexity.
  2. This exposes a(nother) DOS technique, malicious executors can send large JSON blobs frequently.
The slave may have to manage its memory in order to mitigate this.
  3. ?

But, assuming we were to implement such a feature:
This would probably be best as an added call on the ExecutorDriver, allowing executors to
export stats to the slave.

All of the existing Executor interface calls do not return values, and rather the Executor
has to take action by making calls to the driver. Optionally we can add a call to the Executor
interface, asking for json. This would be beneficial for frameworks that want to be polled
for exporting.

Additionally, the slave needs an endpoint for exporting JSON on behalf of the executors. Currently,
the ExecutorDriver (see: exec.cpp) only sends messages to the slave, so the JSON may have
to be passed through the slave for simplicity initially. We probably want to then create a
new libprocess Process to be responsible for this storing / exposing the JSON for each executor.

This is how I would implement this:
1. Add export(json) to the ExecutorDriver.
2. Send the JSON data to the slave through a new message type.
3. Have the slave relay the data to a new libprocess Process that exports the data on a /executors/exported.json
or the like.
> add exportStats method to executor
> ----------------------------------
>                 Key: MESOS-433
>                 URL:
>             Project: Mesos
>          Issue Type: New Feature
>          Components: slave, webui
>            Reporter: brian wickman
> Now that the slave exports per-executor stats via the UI, the slave should be able to
periodically request application-specific stats from the executor (e.g. zookeeper session
id, connection counts, whatever) and export them on the executor's behalf.  This way we don't
need to reinvent any observability wheels.
> The interface could either be exportStats(task_id) or just exportStats(), and return
a JSON object with all stats to export.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message