mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dalton Matos Coelho Barreto (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MESOS-8517) Master http endpoint /tasks should include "executor/directory" field
Date Wed, 31 Jan 2018 14:50:01 GMT

     [ https://issues.apache.org/jira/browse/MESOS-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dalton Matos Coelho Barreto updated MESOS-8517:
-----------------------------------------------
    Description: 
Hello all,

I noticed that the Task information returned by a mesos agent is different from the task information
returned by the mesos master.

Agent's {{/state}} endpoint, when returning each task, includes the {{directory}} field (inside
the {{executors}} array), which is super useful if we want to browse this task's sandbox files.
But at the same time, the agent HTTP API does not provide a way to get a task by its ID.

So we need to talk to the Mesos master to get a task by its ID and we can do it using the
{{/tasks?task_id=<task_id}} endpoint. But here we do not have any information about this
task's executor.

So given a TaskID, to be able to download/browse its sandbox files we need 3+ API calls:
 * One to the master ({{/tasks}}) to get the task's JSON;
 * Anoter to the master ({{/slaves?slave_id=<slave-id>}} to get the slave IP address;
 * Another to this slave ({{//state}}) to get all its tasks. Then we iterate the lists of
tasks (of each executor) searching for our TaskID; get the {{directory}} field.
 * And then, the final API call, made on endpoint ({{/files/browse?path=}}) to get the contents
of the required file

I would like to know if it's possible to the Master to include this {{directory}} field when
returning info about a specific task.

Also, is there a easier way to do what I described here? GIven a TaskID, get the content of
a file inside its sandbox (stdout/stderr would be the most common options)

Thanks a lot.

Here is one example of each JSON returned (only the important parts)

Master {{/tasks?task_id=<task_id>}} endpoint
{noformat}
{
   "tasks" : [
      {
         "resources" : {
            "disk" : 0,
            "gpus" : 0,
            "ports" : "[31720-31720]",
            "cpus" : 0.3,
            "mem" : 256
         },
         "executor_id" : "",
         "labels" : [
         ],
         "framework_id" : "c7fe8caf-3772-4570-bb36-149d600053af-0000",
         "id" : "",
         "slave_id" : "78a58d98-f011-41bf-b5d7-a92700bcda93-S20847",
         "discovery" : { },
         "role" : "*",
         "state" : "TASK_RUNNING"
      }
   ]
}
{noformat}
Slave's {{/state}} endpoint
{noformat}
{
   "build_user" : "centos",
   "frameworks" : [
      {
         "role" : "*",
         "completed_executors" : ,
         "executors" : [
            {
               "tasks" : [ ],
               "directory" : "/tmp/mesos/slaves/78a58d98-f011-41bf-b5d7-a92700bcda93-S20847/frameworks/c7fe8caf-3772-4570-bb36-149d600053af-0000/executors/task_name.447d755d-0616-11e8-96d4-ee61be2ee5b4/runs/14f119de-708c-4978-b04e-60886a12be57",
               "container" : "14f119de-708c-4978-b04e-60886a12be57",
               "queued_tasks" : [],
               "resources" : {
                  "cpus" : 0.4,
                  "mem" : 288,
                  "ports" : "[31720-31720]",
                  "disk" : 0,
                  "gpus" : 0
               },
               "completed_tasks" : [],
               "labels" : [ ],
               "role" : "*"
            }
         ],
         "id" : "c7fe8caf-3772-4570-bb36-149d600053af-0000",
         "checkpoint" : true
      }
   ]
}

{noformat}

Thanks,


  was:
'Hello all,

I noticed that the Task information returned by a mesos agent is different from the task information
returned by the mesos master.

Agent's {{/state}} endpoint, when returning each task, includes the {{directory}} field (inside
the {{executors}} array), which is super useful if we want to browse this task's sandbox files.
But at the same time, the agent HTTP API does not provide a way to get a task by its ID.

So we need to talk to the Mesos master to get a task by its ID and we can do it using the
{{/tasks?task_id=<task_id}} endpoint. But here we do not have any information about this
task's executor.

So given a TaskID, to be able to download/browse its sandbox files we need 3+ API calls:
 * One to the master ({{/tasks}}) to get the task's JSON;
 * Anoter to the master ({{/slaves?slave_id=<slave-id>}} to get the slave IP address;
 * Another to this slave ({{//state}}) to get all its tasks. Then we iterate the lists of
tasks (of each executor) searching for our TaskID; get the {{directory}} field.
 * And then, the final API call, made on endpoint ({{/files/browse?path=}}) to get the contents
of the required file

I would like to know if it's possible to the Master to include this {{directory}} field when
returning info about a specific task.

Also, is there a easier way to do what I described here? GIven a TaskID, get the content of
a file inside its sandbox (stdout/stderr would be the most common options)

Thanks a lot.

Here is one example of each JSON returned (only the important parts)

Master {{/tasks?task_id=<task_id>}} endpoint
{noformat}
{
   "tasks" : [
      {
         "resources" : {
            "disk" : 0,
            "gpus" : 0,
            "ports" : "[31720-31720]",
            "cpus" : 0.3,
            "mem" : 256
         },
         "executor_id" : "",
         "labels" : [
         ],
         "framework_id" : "c7fe8caf-3772-4570-bb36-149d600053af-0000",
         "id" : "",
         "slave_id" : "78a58d98-f011-41bf-b5d7-a92700bcda93-S20847",
         "discovery" : { },
         "role" : "*",
         "state" : "TASK_RUNNING"
      }
   ]
}
{noformat}
Slave's {{/state}} endpoint
{noformat}
{
   "build_user" : "centos",
   "frameworks" : [
      {
         "role" : "*",
         "completed_executors" : ,
         "executors" : [
            {
               "tasks" : [ ],
               "directory" : "/tmp/mesos/slaves/78a58d98-f011-41bf-b5d7-a92700bcda93-S20847/frameworks/c7fe8caf-3772-4570-bb36-149d600053af-0000/executors/task_name.447d755d-0616-11e8-96d4-ee61be2ee5b4/runs/14f119de-708c-4978-b04e-60886a12be57",
               "container" : "14f119de-708c-4978-b04e-60886a12be57",
               "queued_tasks" : [],
               "resources" : {
                  "cpus" : 0.4,
                  "mem" : 288,
                  "ports" : "[31720-31720]",
                  "disk" : 0,
                  "gpus" : 0
               },
               "completed_tasks" : [],
               "labels" : [ ],
               "role" : "*"
            }
         ],
         "id" : "c7fe8caf-3772-4570-bb36-149d600053af-0000",
         "checkpoint" : true
      }
   ]
}

{noformat}

Thanks,



> Master http endpoint /tasks should include "executor/directory" field
> ---------------------------------------------------------------------
>
>                 Key: MESOS-8517
>                 URL: https://issues.apache.org/jira/browse/MESOS-8517
>             Project: Mesos
>          Issue Type: Improvement
>          Components: HTTP API
>    Affects Versions: 1.4.1
>            Reporter: Dalton Matos Coelho Barreto
>            Priority: Minor
>
> Hello all,
> I noticed that the Task information returned by a mesos agent is different from the task
information returned by the mesos master.
> Agent's {{/state}} endpoint, when returning each task, includes the {{directory}} field
(inside the {{executors}} array), which is super useful if we want to browse this task's sandbox
files. But at the same time, the agent HTTP API does not provide a way to get a task by its
ID.
> So we need to talk to the Mesos master to get a task by its ID and we can do it using
the {{/tasks?task_id=<task_id}} endpoint. But here we do not have any information about
this task's executor.
> So given a TaskID, to be able to download/browse its sandbox files we need 3+ API calls:
>  * One to the master ({{/tasks}}) to get the task's JSON;
>  * Anoter to the master ({{/slaves?slave_id=<slave-id>}} to get the slave IP address;
>  * Another to this slave ({{//state}}) to get all its tasks. Then we iterate the lists
of tasks (of each executor) searching for our TaskID; get the {{directory}} field.
>  * And then, the final API call, made on endpoint ({{/files/browse?path=}}) to get the
contents of the required file
> I would like to know if it's possible to the Master to include this {{directory}} field
when returning info about a specific task.
> Also, is there a easier way to do what I described here? GIven a TaskID, get the content
of a file inside its sandbox (stdout/stderr would be the most common options)
> Thanks a lot.
> Here is one example of each JSON returned (only the important parts)
> Master {{/tasks?task_id=<task_id>}} endpoint
> {noformat}
> {
>    "tasks" : [
>       {
>          "resources" : {
>             "disk" : 0,
>             "gpus" : 0,
>             "ports" : "[31720-31720]",
>             "cpus" : 0.3,
>             "mem" : 256
>          },
>          "executor_id" : "",
>          "labels" : [
>          ],
>          "framework_id" : "c7fe8caf-3772-4570-bb36-149d600053af-0000",
>          "id" : "",
>          "slave_id" : "78a58d98-f011-41bf-b5d7-a92700bcda93-S20847",
>          "discovery" : { },
>          "role" : "*",
>          "state" : "TASK_RUNNING"
>       }
>    ]
> }
> {noformat}
> Slave's {{/state}} endpoint
> {noformat}
> {
>    "build_user" : "centos",
>    "frameworks" : [
>       {
>          "role" : "*",
>          "completed_executors" : ,
>          "executors" : [
>             {
>                "tasks" : [ ],
>                "directory" : "/tmp/mesos/slaves/78a58d98-f011-41bf-b5d7-a92700bcda93-S20847/frameworks/c7fe8caf-3772-4570-bb36-149d600053af-0000/executors/task_name.447d755d-0616-11e8-96d4-ee61be2ee5b4/runs/14f119de-708c-4978-b04e-60886a12be57",
>                "container" : "14f119de-708c-4978-b04e-60886a12be57",
>                "queued_tasks" : [],
>                "resources" : {
>                   "cpus" : 0.4,
>                   "mem" : 288,
>                   "ports" : "[31720-31720]",
>                   "disk" : 0,
>                   "gpus" : 0
>                },
>                "completed_tasks" : [],
>                "labels" : [ ],
>                "role" : "*"
>             }
>          ],
>          "id" : "c7fe8caf-3772-4570-bb36-149d600053af-0000",
>          "checkpoint" : true
>       }
>    ]
> }
> {noformat}
> Thanks,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message