Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAE=7Uom-obUnZ5bT1eabEkO8jt=bLisjZ-xho_RPgEgFbkghVg@mail.gmail.com>
References: 
 <CADq0cj7zFtURJQYKez-K-ROxKG6JR4NTSjJdQ+51D+tOWfB6yA@mail.gmail.com>
	<CAOFpRGBukLrvAbaFnn-4Kc-9KONEpsYAXE=2FyRK4_TY14w3xw@mail.gmail.com>
	<CADq0cj6mNuUAkm+y4mncWD5iPNh9Osd+Cb2=BrX6BDrVR6Hkhg@mail.gmail.com>
	<567D71A9.9020804@weimo.de>
	<CAE=7UomgPAkuB4MN_v+1rCLiMSexcVwbRRAk1BY=qKR_Yjbh-g@mail.gmail.com>
	<5684DF0D.6090800@weimo.de>
	<CAE=7UonbCh_Emg2j4Uxra1UeBR-BFqdMgTg1rtGAKQWCtZc6jA@mail.gmail.com>
	<56856560.2000803@weimo.de>
	<CAE=7UokYecBcmyaMTT1-3VKngKqODoM8RyTod5JLpsBW4xDcog@mail.gmail.com>
	<CAOFpRGB0uHqxc1MmEj12GAZLHH2Y5Pm6oKYhT_UWNgdmvmXAYQ@mail.gmail.com>
	<CADq0cj7GmXoPciEBjGFNnMCK4wr2cFtERJ50otk7Qfa0PXFKPw@mail.gmail.com>
	<CAE=7Uom-obUnZ5bT1eabEkO8jt=bLisjZ-xho_RPgEgFbkghVg@mail.gmail.com>
Date: Thu, 7 Jan 2016 18:14:01 +0900
Message-ID: 
 <CADq0cj6aXzp_BP+G0_v6HJEfG3neZfpUB7no9OwfQXAk__+umA@mail.gmail.com>
Subject: Re: new runtime: stand-alone distributed runtime?
From: Byung-Gon Chun <bgchun@gmail.com>
To: dev@reef.apache.org
Content-Type: multipart/alternative; boundary=94eb2c086ea034a4600528bae43b

--94eb2c086ea034a4600528bae43b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks, John.
I will assign it to Dongjin, our new intern.

-Gon

On Mon, Jan 4, 2016 at 1:50 PM, John Yang <johnyangk@gmail.com> wrote:

> I opened a JIRA:
> https://issues.apache.org/jira/browse/REEF-1123
>
> Thanks,
> John
>
> On Fri, Jan 1, 2016 at 4:48 PM, Byung-Gon Chun <bgchun@gmail.com> wrote:
>
> > Thanks for the input, Markus!
> > John, can you create an issue regarding the stand-alone distributed
> runtime
> > and assign it to DongJin?
> >
> > -Gon
> >
> > On Fri, Jan 1, 2016 at 4:26 PM, Markus Weimer <markus@weimo.de> wrote:
> >
> > > Great! I think it is time to move the remainder of the design and
> > > implementation discussions onto a JIRA?
> > >
> > > Markus
> > >
> > > On Thu, Dec 31, 2015 at 4:53 PM, John Yang <johnyangk@gmail.com>
> wrote:
> > > > Cool! Your suggestions sound great. I'll help the intern in
> > implementing
> > > it.
> > > >
> > > > Thanks,
> > > > John
> > > > 2016. 1. 1. =EC=98=A4=EC=A0=84 2:27=EC=97=90 "Markus Weimer" <marku=
s@weimo.de>=EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=B1:
> > > >
> > > >> On 2015-12-31 00:14, John Yang wrote:
> > > >>
> > > >>> So please allow me to ask some more questions.
> > > >>>
> > > >>
> > > >> Go ahead. But understand that I've not given this nearly as much
> > thought
> > > >> as you :-)
> > > >>
> > > >> - In which directory to scp the jars/resources? - /tmp comes to my
> > > >>> mind
> > > >>>
> > > >>
> > > >> We could make that configurable, and default to something like
> > > >> `./REEF_SSH/` with the same naming scheme as for the local runtime=
.
> > > >> Using a relative path puts it into the user's $HOME directory for
> easy
> > > >> access. And using the local runtime's file system layout ensures
> that
> > in
> > > >> the (common) case of NFS-mounted $HOME, we don't produce name
> clashes.
> > > >>
> > > >> Actually, for NFS mounted $HOME, we could even optimize the `scp`
> > round
> > > >> away. But that is for later stages :)
> > > >>
> > > >> - How shall we retrieve the Evaluator logs? - Somehow redirect
> > > >>> stdout/stderr through the SSH connection to the Driver to be stor=
ed
> > > >>> on its node
> > > >>>
> > > >>
> > > >> That would put a lot of load on the Driver machine's network
> > connection.
> > > >> On the plus side, it makes the logs instantly available.
> > > >>
> > > >> Another alternative, that reverses these benefits, would be to wri=
te
> > the
> > > >> logs local on the Evaluator side, and then scp them to the Driver =
at
> > the
> > > >> end. And from there, we could scp them to the Client after the
> Driver
> > > >> exits. This has longer waits for the logs to be available, but is
> > likely
> > > >> more robust and efficient.
> > > >>
> > > >> - How shall we teach the SSH runtime the available hosts? - Maybe =
we
> > > >>> allow Tang configuration of List<String> that represent a list of
> ip
> > > >>> addresses
> > > >>>
> > > >>
> > > >> I'd use a `Set`, as the order likely doesn't matter. We also need =
a
> > > >> username for the ssh connections. For a first cut, I believe it is
> OK
> > > >> for us to assume that public keys have been set up for
> authentication.
> > > >>
> > > >> Markus
> > > >>
> > >
> >
> >
> >
> > --
> > Byung-Gon Chun
> >
>


--=20
Byung-Gon Chun

--94eb2c086ea034a4600528bae43b--