Mailing-List: contact mesos-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mesos-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of benjamin.mahler@gmail.com
 designates 209.85.212.51 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPXBsRDgHt6k-zfiQdBc4fy_jBvwL_QE-t8srN___whueHKZUA@mail.gmail.com>
References: 
 <CAPXBsRAX1WOeMfaYLWs2hkHGb54freKzCrjvp3+PXkx5qe5tXA@mail.gmail.com>
 <CA+8RcoQJwBNaki_Xw4ox5WOx+Aq54TLar3ApjDTqOaEhO2uUiA@mail.gmail.com>
 <CAPXBsRDhen68GgvV8KDnGjPUJFNPic71bpedFh9h2G7r_T5v9g@mail.gmail.com>
 <CAPXBsRDgHt6k-zfiQdBc4fy_jBvwL_QE-t8srN___whueHKZUA@mail.gmail.com>
From: Benjamin Mahler <benjamin.mahler@gmail.com>
Date: Wed, 10 Apr 2013 09:58:29 -0700
Message-ID: 
 <CA+8RcoRS-OcVP8b6E-qFXXDv8AvV-KaeDgWQNB4DgbK2bfJB5w@mail.gmail.com>
Subject: Re: long lived framework can not launch task
To: "mesos-dev@incubator.apache.org" <mesos-dev@incubator.apache.org>
Content-Type: multipart/alternative; boundary=089e011822e884770a04da049542

--089e011822e884770a04da049542
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Yes, each executor ID can have only one 'source'. That is the only
constraint, which means you can share the same 'source' across different
executors. To see the benefit of the 'source' field, take this example
below.

Say we have a framework and we want to spin up three executors to run a
job. We can have three executors:
1: ExecutorID: "e_1", source: "page_rank_job"
1: ExecutorID: "e_2", source: "page_rank_job"
1: ExecutorID: "e_3", source: "page_rank_job"

This allows us to identify the source of the executor. This is also
exported as part of the monitoring stats on each slave.

$ curl <slave>:5051/monitor/usage.json
[
  {"executor_id":"e1",
   "executor_name": "task_tracker",
   "source": "page_rank_job",
   "framework_id":"201301240028-2449582272-5050-61291-0000",
   "resource_usage":
     {"cpu_time":1.0,       // CPU time (user + system) in seconds.
      "cpu_usage":0.5,      // CPU usage [0.5 ~ 50% of 1 core].
      "memory_rss":1048576, // RSS in bytes.
      // ... Additional stats in the future.
     },
  },
  {"executor_id":"e2",
   "executor_name": "task_tracker",
   "source": "page_rank_job",
   "framework_id":"201301240028-2449582272-5050-61291-0000",
   "resource_usage":
     {"cpu_time":1.0,       // CPU time (user + system) in seconds.
      "cpu_usage":16.0,     // CPU usage [16.0 ~ 1600% usage ~ 100% of 16 c=
ores]
      "memory_rss":1048576, // RSS in bytes.
      // ... Additional stats in the future.
     },
   },


Here we can aggregate all the resource usage for the page_rank_job by
asking all of the slaves for the usage information. But if you're looking
to write a framework, you can do without setting the source field for now,
and set it later if the need arises.

I'll eventually have some more documentation around this:
https://issues.apache.org/jira/browse/MESOS-373

On Wed, Apr 10, 2013 at 2:28 AM, =CD=F5=B9=FA=B6=B0 <wanggd04@gmail.com> wr=
ote:

> Hi Ben,
>
> I find some comment in mesos.proto file,
>   // Source is an identifier style string used by frameworks to track
>   // the source of an executor.
>
> Does this mean each executorId can only have the identical source ?
> In the old long-lived-framework, we are trying to assign different source=
s
> to the same executor. So the error happens.
>
> Am I right?
>
> Thanks.
>
> Guodong
>
>
> On Wed, Apr 10, 2013 at 3:02 PM, =CD=F5=B9=FA=B6=B0 <wanggd04@gmail.com> =
wrote:
>
> > Hi Ben,
> >
> > It works now. Thank you for your reply.
> >
> > I am trying to learn to write a framework on mesos. But I can not find
> the
> > exact meaning of each param in the API. eg. I do not know the meaning o=
f
> > "source".  Where can I find the docs about the programming guide?
> >
> > Thanks.
> >
> >
> > Guodong
> >
> >
> > On Wed, Apr 10, 2013 at 2:09 AM, Benjamin Mahler <
> > benjamin.mahler@gmail.com> wrote:
> >
> >> Thanks for the report!
> >>
> >> The bug here is that we set the source for each task:
> >>
> >>         TaskInfo task;
> >>         task.set_name("Task " + lexical_cast<string>(taskId));
> >>         task.mutable_task_id()->set_value(lexical_cast<string>(taskId)=
);
> >>         task.mutable_slave_id()->MergeFrom(offer.slave_id());
> >>         task.mutable_executor()->MergeFrom(executor);
> >>         *task.mutable_executor()->set_source("task_" +
> >> stringify(taskId));*
> >>
> >> I'll have a review out shortly to fix this.
> >>
> >>
> >> On Tue, Apr 9, 2013 at 2:43 AM, =CD=F5=B9=FA=B6=B0 <wanggd04@gmail.com=
> wrote:
> >>
> >> > hi
> >> >
> >> > I am trying to run long-lived-framework in the trunk.
> >> > But I find the following error after task 1 finished.
> >> >
> >> > W0409 17:18:03.841472 15305 master.cpp:1566] Error validating task 1=
 :
> >> Task
> >> > has invalid ExecutorInfo (existing ExecutorInfo with same ExecutorID
> is
> >> not
> >> > compatible)
> >> >
> >> > Then all the tasks will be lost.
> >> >
> >> > The log of the framework is as follow:
> >> > Registered!
> >> > .Starting task 0 on guodong-Vostro-3400
> >> > Task 0 is in state 1
> >> > Task 0 is in state 2
> >> > .Starting task 1 on guodong-Vostro-3400
> >> > Task 1 is in state 5
> >> > .Starting task 2 on guodong-Vostro-3400
> >> > Task 2 is in state 5
> >> > .Starting task 3 on guodong-Vostro-3400
> >> > Task 3 is in state 5
> >> >
> >> >
> >> > I also go through the code of LongLivedFramework. And I can not
> >> understand
> >> > this error, since the ExecutorInfo is passed as the constructor
> >> arguments
> >> > of Scheduler.
> >> >
> >> > Best regards.
> >> >
> >> > Guodong
> >> >
> >>
> >
> >
>

--089e011822e884770a04da049542--