Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
MIME-Version: 1.0
In-Reply-To: <CAE=7UokxnUGe+WTJ4Zsj8+eg6mK8XkR39akZBxwq_xy07xHNbg@mail.gmail.com>
References: <SN2PR03MB2381E57990770B4DBF996AF3902C0@SN2PR03MB2381.namprd03.prod.outlook.com>
 <CAPCBsnNHwYKgFfK6OB6=UMT_MNf+jMHG15-JLaF+0XhimgcB7A@mail.gmail.com>
 <5f4ae285-bbbc-6050-9931-93185fd68a60@weimo.de> <CAE=7UokxnUGe+WTJ4Zsj8+eg6mK8XkR39akZBxwq_xy07xHNbg@mail.gmail.com>
From: John Yang <johnyangk@gmail.com>
Date: Wed, 22 Jun 2016 23:38:03 -0600
Message-ID: <CAE=7UomNGvQXe58Xe8cqCqOm68SmKZUxg7eTYdg6inRrxMAVTw@mail.gmail.com>
Subject: Re: Question about failure scenarios
To: dev@reef.apache.org
Content-Type: multipart/alternative; boundary=001a11411f28440f180535eb7510
archived-at: Thu, 23 Jun 2016 05:38:11 -0000

--001a11411f28440f180535eb7510
Content-Type: text/plain; charset=UTF-8

Julia, with regard to the REEF Task submission, here's how Vortex goes on
about it.

- Upon AllocatedEvaluatorHandler, Vortex submits a REEF Task without any
bookkeeping. (Note that Vortex's REEF Task itself does not do anything
useful itself upon instantiation; It only waits for the Driver to schedule
tasklets onto it.)
- *(If FailedEvaluatorHandler is invoked here)* We request for a new
Evaluator. FailedEvaluator#getFailedTask returns null, and is thus ignored.
- RunningTaskHandler is invoked and we memorize the task's id. (At this
point of time, Vortex considers the Evaluator to be ready have tasklets
scheduled onto)
- *(If FailedEvaluatorHandler is invoked here) *We request for a
new Evaluator and obtain the task id from FailedEvaluator#getFailedTask to
un-memorize it

Note that we rely on the fact that Driver uses a dedicated thread per
Evaluator for processing their incoming events, which guarantees the
invocations of their handlers to be serialized. Because of this, I believe
RunningTaskHandler is never invoked after FailedEvaluatorHandler, and thus
no garbage task ids will be created.


Thanks,
John


On Wed, Jun 22, 2016 at 10:42 PM, John Yang <johnyangk@gmail.com> wrote:

> Hi,
>
>
> Things are the same with the Mesos runtime. It receives resource offers
> from Mesos. If the offers do not satisfy the resource requests from the
> REEF user, it declines the offers, in order to receive new offers. It keeps
> at it until it gets the right offers, which might never come.
>
> The best solution I can think of right now is similar to Markus's. You
> have to somehow identify this behavior at the REEF-user-level based on your
> application semantics. If we were to have this functionality at the RM
> runtime-level(reef-runtime-yarn, reef-runtime-mesos, etc), I guess we can
> make a set of standard configuration parameters to be used in each runtime.
> However I'm not sure the benefits outweigh the added complexity, since the
> resource capacity/availability of a cluster is usually considered
> non-deterministic.
>
>
> Thanks,
> John
>
>
> On Wed, Jun 22, 2016 at 2:58 PM, Markus Weimer <markus@weimo.de> wrote:
>
>> On 2016-06-22 13:31, Tobin Baker wrote:
>>
>>> My experience on the Java side has been that if insufficient
>>> resources are available to allocate all requested Evaluators, the RM
>>> will just silently keep retrying forever, with no exceptions thrown
>>> by YARN or REEF.
>>>
>>
>> That is correct. YARN does not, presently, give "no" as an answer to a
>> unsatisfiable resource request. The only way I know to guard against it
>> is to set a timer and to give up if the needed containers can't be
>> acquired within the timeout.
>>
>> Markus
>>
>
>

--001a11411f28440f180535eb7510--