reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Kanjilal <sxk1...@gmail.com>
Subject Re: REEF scheduler: Tasks groups and fault tolerance
Date Thu, 13 Jul 2017 23:42:38 GMT
Hey Matteo,
Does it make sense to do a pull request with a design doc embedded, it'd be
easier to review and get it into reef at some point?
Thanks

On Thu, Jul 13, 2017 at 4:36 PM, Matteo Interlandi <m.interlandi@gmail.com>
wrote:

> For the interested people, I have update the current code I have in the
> elastic branch of my local REEF fork [1]. Most of the code is
> under Org.Apache.REEF.Network/Elastic.
>
> [1] https://github.com/interesaaat/reef/tree/elastic
>
> On Thu, Jul 13, 2017 at 10:09 AM, Matteo Interlandi <
> m.interlandi@gmail.com>
> wrote:
>
> > I think at the end we may need that. I am not focusing on these issues
> for
> > the moment because we are targeting iterative (single stage) kind of
> > applications where you almost have a 1 to 1 mapping between Tasks,
> Contexts
> > and Evaluators. In general however, you will have a dataflow of TaskSets,
> > where on TaskSet complete, Evaluators and Contexts are handed over other
> > TaskSets, etc.
> >
> > We may also need them for failure logic, but I am not sure at this point.
> > I think that I will have a clearer idea when I will start to runt he code
> > in the wild with real errors.
> >
> > --Matteo
> >
> > On Thu, Jul 13, 2017 at 9:44 AM, Markus Weimer <markus@weimo.de> wrote:
> >
> >> Thanks, Matteo for sharing!
> >>
> >> One question I have is how to deal with Evaluators in addition to
> >> Tasks. One key value prop in REEF is that Evaluators and Tasks have
> >> independent life cycles. Would we have a similar concept of
> >> `EvaluatorSets`? Or more likely, `ContextSets`?
> >>
> >> Thanks,
> >>
> >> Markus
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message