Mailing-List: contact mesos-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mesos-dev@incubator.apache.org
Content-Type: multipart/alternative;
	boundary="===============4158697575398700982=="
MIME-Version: 1.0
Subject: Re: Review Request: Basic virtual machine isolation module for Mesos
From: "Charles Reiss" <woggling@gmail.com>
To: "Benjamin Hindman" <benh@berkeley.edu>, "Matei Zaharia" <matei@apache.org>
Date: Tue, 17 Apr 2012 19:50:41 -0000
Message-ID: <20120417195041.6555.13674@reviews.apache.org>
Cc: "Charles Earl" <charles.cearl@gmail.com>,
	"mesos" <mesos-dev@incubator.apache.org>,
	"Charles Reiss" <woggling@gmail.com>
In-Reply-To: <20120414232721.17356.98241@reviews.apache.org>
References: <20120414232721.17356.98241@reviews.apache.org>

--===============4158697575398700982==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable


> On 2012-04-14 23:27:22, Charles Reiss wrote:
> > I'd prefer we avoid this approach where we need to discover the VM IP a=
nd SSH into it. Can we instead pass boot options and have a startup script =
on the VM launch something that reads a protobuf contact address out of the=
 boot options and retrieves a (protobuf-format) description of what to laun=
ch? [This should also avoid fragility from generating a shell script.]
> > =

> > On a lower priority, there are some hard-coded paths and some use of me=
sos_home that won't work; we need to handle (a) out-of-source builds and (b=
) installation to "standard" paths ($(prefix)/bin $(prefix)/libexec/mesos, =
etc.).
> > =

> >
> =

> Charles Earl wrote:
>     Yes, I agree that hard coded paths should be dispensed with and will =
address those shortly. If I understand the main point, a better solution is=
 to pass the VM a fixed IP address as a boot parameter. Using this address =
the VM can then provide its address and also receive launch configuration. =
This eliminates the need to scp the shell script and also eliminates need f=
or find_addr.pl etc.
> =

> Charles Earl wrote:
>     Further, I would assume that mesos.conf is where configuration option=
s should (e.g. "phone home" address, options for the VM) placed.

What probably should be passed is the stringified form of a process::UPID f=
or a process::Process<XXX> subclass managed by the isolation module along w=
ith the framework+executor ID (or an equivalent token). The VM's init scrip=
ts will then extract this from /proc/cmdline (for Linux VMs) and run some (=
new?) wrapper similar to mesos-launcher to phone home.


- Charles


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4717/#review6925
-----------------------------------------------------------


On 2012-04-14 03:57:12, Charles Earl wrote:
> =

> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4717/
> -----------------------------------------------------------
> =

> (Updated 2012-04-14 03:57:12)
> =

> =

> Review request for mesos, Benjamin Hindman and Matei Zaharia.
> =

> =

> Summary
> -------
> =

> Earlier in the year I implemented a virtual machine isolation module. Thi=
s module uses lib-virt to launch and manage virtual machine containers. The=
 code is still rough and have done basic testing with the Spark example. =

> =

> This code works with the KVM (http://www.linux-kvm.org/page/Main_Page) vi=
rtual machine manager. I've placed the relevant code in a branch called mes=
os-vm, for now located at https://github.com/charlescearl/VirtualMesos. The=
 code is based upon the mesos lxc isolation module that is located in src/s=
lave/lxc_isolation_module.cpp/.hpp. My code based on the mesos master branc=
h dated Wed Nov 23 12:02:07 2011 -0800, commit 059aabb2ec5bd7b20ed08ab9c439=
531a352ba3ec. I've included a patch for the relevant code included for the =
review. Suggestions appreciated on whether this is the appropriate branch/c=
ommit to patch against. =

> =

> Most of the implementation is contained in vm_isolation_module.cpp and vm=
_isolation_module.hpp and there are some minor additions in launcher to han=
dle setup of the environment for the virtual machine. I use the libvirt (ht=
tp://libvirt.org/) library, to manage the virtual machine container in whic=
h the jobs are executed. =

> =

> Dependencies =

> The code has been tested on Ubuntu 11.04 and 11.10 and depends on libpyth=
on2.6 and libvirt0 =

> =

> Configuration of the virtual machine container =

> The virtual machine invocation depends upon a few configuration assumptio=
ns: =

>  1. ssh public keys installed on the container. I assume that the contain=
er is setup to allow password-less secure access. =

>   2. Directory structure on the container matches the servant machine. Fo=
r example, in invoking a spark executor, assume that the paths match the se=
tup on the container host. =

> =

> Running it =

> In the $MESOS_HOME/conf/mesos.conf file add the line =

>    isolation=3Dvm =

> to use the virtual machine isolation. =

> =

> The Mesos slave is invoked with the isolation parameter set to vm. For ex=
ample =

>      sudo bin/mesos-slave -m mesos://master@mesos-host:5050 -w 9839 --iso=
lation=3Dvm =

> =

> Rough description of how it works =

> =

> The `vm_isolation_module` class forks a process that in turn launches a v=
irtual machine. A routine located in bin called find_addr.pl is responsible=
 for figuring out the IP address of the launched virtual machine. This is p=
robably not portable since it is explicitly looking for entry in the virbr0=
 network. =

> =

> A script vmLauncherTemplate.sh located in bin assists the the vmLauncher =
method to setup the environment for launching tasks inside of the virtual m=
achine. The vmLauncher method uses vmLauncherTemplate.sh to create a tasks =
specific shell vmLauncherTemplate-<task_id>.sh, which is copied to the runn=
ing guest and used to run the executor inside the VM. This communicates wit=
h the slave on the host. =

> =

> Comments and suggestions on improvements and next directions are apprecia=
ted!
> =

> =

> Diffs
> -----
> =

>   bin/find_addr.pl PRE-CREATION =

>   bin/killtree.sh PRE-CREATION =

>   bin/vmLauncher.sh PRE-CREATION =

>   bin/vmLauncherTemplate.sh PRE-CREATION =

>   src/config/config.hpp PRE-CREATION =

>   src/launcher/launcher.hpp b99b6d2 =

>   src/launcher/launcher.cpp 4422224 =

>   src/launcher/vm_mesos_launcher.cpp PRE-CREATION =

>   src/slave/isolation_module.cpp 5b7b4a2 =

>   src/slave/isolation_module_factory.cpp 6498945 =

>   src/slave/lxc_isolation_module.cpp ab0843a =

>   src/slave/main.cpp 9519ed2 =

>   src/slave/slave.cpp 21fc9f2 =

>   src/slave/vm_isolation_module.hpp PRE-CREATION =

>   src/slave/vm_isolation_module.cpp PRE-CREATION =

> =

> Diff: https://reviews.apache.org/r/4717/diff
> =

> =

> Testing
> -------
> =

> This was run with the spark example on single KVM virtual machine. Not te=
sted extensively.
> =

> =

> Thanks,
> =

> Charles
> =

>


--===============4158697575398700982==--