reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Weimer <mar...@weimo.de>
Subject Re: Introduction and question about REEF local mode
Date Thu, 30 Mar 2017 21:43:34 GMT
Welcome to the list, Matteo!

Sergiy has done some work to make it so that Driver and Client can be in
the same process. That way, you can easily step through your Driver in a
debugger, even if the Evaluators are running on a YARN cluster. @Sergiy:
Maybe it is time to have a short documentation of how to do that?

Regarding Evaluators in the same process: We used to have this, and the
local runtime still has vestiges of that support. The `ResourceManager` in
the local runtime uses the `ContainerManager` to ultimately launch the
Evaluators. `ContainerManager` is presently hard coded to create
`ProcessContainers` (see line 372). In order to create in-process, let's
call them `ThreadContainers`, that code needs to be made configurable.
Maybe we need a new injectable interface `ContainerFactory` with
implementations for `ProcessContainer` and `ThreadContainer` which we can
use in that line to make a new `Container` instance?

Markus



On Thu, Mar 30, 2017 at 11:44 AM, Matteo Interlandi <m.interlandi@gmail.com>
wrote:

> Hi REEF developers,
>
> my name is Matteo Interlandi and I am currently postdoc at UCLA. I am
> getting familiar with REEF and I was noticing that in local mode different
> processes for the driver and evaluators are spawned. While this is good for
> testing, I found a little bit difficult to do debugging and understanding
> the flow of execution. I was wondering if it could be useful to add a
> thread-based local mode, as in Apache Spark. Of course the thread and
> process modes code paths should be as overlapping as possible to help
> testing.
>
> I have an hack working on java but I would like to know from you what
> should be the clean (REEF) way of doing this, or if you already have
> something similar there.
>
> Thanks.
> --Matteo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message