reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Weimer <>
Subject Re: new runtime: stand-alone distributed runtime?
Date Fri, 25 Dec 2015 16:41:13 GMT
On 2015-12-24 23:22, Byung-Gon Chun wrote:
> Markus, can you elaborate the SSH story?

Sure. When we launch the Driver in this new runtime, that can happen on 
the machine where the job is launched, just like in the local runtime. 
My question revolves around how to get the Evaluators to launch on 
another machine. I can imagine two ways of doing this:

   (1) REEF gets an agent that needs to be deployed on all the nodes you 
may want to target. That agent performs some of the same functionality 
of YARN's NodeManager, but can be simplified a lot as we are not after 
resource and security isolation here. Further, it could receive the ZIP 
containing the Evaluator's resources.

   (2) We use an agent already on the machines. On linux (and soon 
Windows), SSH comes to mind serving that need. We can `scp` the needed 
files to the node where we want to run and then use `ssh` to unzip / 
launch it. If we go down this route, we should make the actual commands 
plugable such that other remoting tools can be used easily.

Currently, I favor option 2. It means less code for us to write but also 
less work for users to deploy this.


View raw message