reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Byung-Gon Chun <bgc...@gmail.com>
Subject Re: new runtime: stand-alone distributed runtime?
Date Thu, 07 Jan 2016 09:14:01 GMT
Thanks, John.
I will assign it to Dongjin, our new intern.

-Gon

On Mon, Jan 4, 2016 at 1:50 PM, John Yang <johnyangk@gmail.com> wrote:

> I opened a JIRA:
> https://issues.apache.org/jira/browse/REEF-1123
>
> Thanks,
> John
>
> On Fri, Jan 1, 2016 at 4:48 PM, Byung-Gon Chun <bgchun@gmail.com> wrote:
>
> > Thanks for the input, Markus!
> > John, can you create an issue regarding the stand-alone distributed
> runtime
> > and assign it to DongJin?
> >
> > -Gon
> >
> > On Fri, Jan 1, 2016 at 4:26 PM, Markus Weimer <markus@weimo.de> wrote:
> >
> > > Great! I think it is time to move the remainder of the design and
> > > implementation discussions onto a JIRA?
> > >
> > > Markus
> > >
> > > On Thu, Dec 31, 2015 at 4:53 PM, John Yang <johnyangk@gmail.com>
> wrote:
> > > > Cool! Your suggestions sound great. I'll help the intern in
> > implementing
> > > it.
> > > >
> > > > Thanks,
> > > > John
> > > > 2016. 1. 1. 오전 2:27에 "Markus Weimer" <markus@weimo.de>님이
작성:
> > > >
> > > >> On 2015-12-31 00:14, John Yang wrote:
> > > >>
> > > >>> So please allow me to ask some more questions.
> > > >>>
> > > >>
> > > >> Go ahead. But understand that I've not given this nearly as much
> > thought
> > > >> as you :-)
> > > >>
> > > >> - In which directory to scp the jars/resources? - /tmp comes to my
> > > >>> mind
> > > >>>
> > > >>
> > > >> We could make that configurable, and default to something like
> > > >> `./REEF_SSH/` with the same naming scheme as for the local runtime.
> > > >> Using a relative path puts it into the user's $HOME directory for
> easy
> > > >> access. And using the local runtime's file system layout ensures
> that
> > in
> > > >> the (common) case of NFS-mounted $HOME, we don't produce name
> clashes.
> > > >>
> > > >> Actually, for NFS mounted $HOME, we could even optimize the `scp`
> > round
> > > >> away. But that is for later stages :)
> > > >>
> > > >> - How shall we retrieve the Evaluator logs? - Somehow redirect
> > > >>> stdout/stderr through the SSH connection to the Driver to be stored
> > > >>> on its node
> > > >>>
> > > >>
> > > >> That would put a lot of load on the Driver machine's network
> > connection.
> > > >> On the plus side, it makes the logs instantly available.
> > > >>
> > > >> Another alternative, that reverses these benefits, would be to write
> > the
> > > >> logs local on the Evaluator side, and then scp them to the Driver
at
> > the
> > > >> end. And from there, we could scp them to the Client after the
> Driver
> > > >> exits. This has longer waits for the logs to be available, but is
> > likely
> > > >> more robust and efficient.
> > > >>
> > > >> - How shall we teach the SSH runtime the available hosts? - Maybe
we
> > > >>> allow Tang configuration of List<String> that represent
a list of
> ip
> > > >>> addresses
> > > >>>
> > > >>
> > > >> I'd use a `Set`, as the order likely doesn't matter. We also need
a
> > > >> username for the ssh connections. For a first cut, I believe it is
> OK
> > > >> for us to assume that public keys have been set up for
> authentication.
> > > >>
> > > >> Markus
> > > >>
> > >
> >
> >
> >
> > --
> > Byung-Gon Chun
> >
>



-- 
Byung-Gon Chun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message