mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Charles Reiss" <woggl...@gmail.com>
Subject Re: Review Request: Basic virtual machine isolation module for Mesos
Date Tue, 17 Apr 2012 19:50:41 GMT


> On 2012-04-14 23:27:22, Charles Reiss wrote:
> > I'd prefer we avoid this approach where we need to discover the VM IP and SSH into
it. Can we instead pass boot options and have a startup script on the VM launch something
that reads a protobuf contact address out of the boot options and retrieves a (protobuf-format)
description of what to launch? [This should also avoid fragility from generating a shell script.]
> > 
> > On a lower priority, there are some hard-coded paths and some use of mesos_home
that won't work; we need to handle (a) out-of-source builds and (b) installation to "standard"
paths ($(prefix)/bin $(prefix)/libexec/mesos, etc.).
> > 
> >
> 
> Charles Earl wrote:
>     Yes, I agree that hard coded paths should be dispensed with and will address those
shortly. If I understand the main point, a better solution is to pass the VM a fixed IP address
as a boot parameter. Using this address the VM can then provide its address and also receive
launch configuration. This eliminates the need to scp the shell script and also eliminates
need for find_addr.pl etc.
> 
> Charles Earl wrote:
>     Further, I would assume that mesos.conf is where configuration options should (e.g.
"phone home" address, options for the VM) placed.

What probably should be passed is the stringified form of a process::UPID for a process::Process<XXX>
subclass managed by the isolation module along with the framework+executor ID (or an equivalent
token). The VM's init scripts will then extract this from /proc/cmdline (for Linux VMs) and
run some (new?) wrapper similar to mesos-launcher to phone home.


- Charles


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4717/#review6925
-----------------------------------------------------------


On 2012-04-14 03:57:12, Charles Earl wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4717/
> -----------------------------------------------------------
> 
> (Updated 2012-04-14 03:57:12)
> 
> 
> Review request for mesos, Benjamin Hindman and Matei Zaharia.
> 
> 
> Summary
> -------
> 
> Earlier in the year I implemented a virtual machine isolation module. This module uses
lib-virt to launch and manage virtual machine containers. The code is still rough and have
done basic testing with the Spark example. 
> 
> This code works with the KVM (http://www.linux-kvm.org/page/Main_Page) virtual machine
manager. I've placed the relevant code in a branch called mesos-vm, for now located at https://github.com/charlescearl/VirtualMesos.
The code is based upon the mesos lxc isolation module that is located in src/slave/lxc_isolation_module.cpp/.hpp.
My code based on the mesos master branch dated Wed Nov 23 12:02:07 2011 -0800, commit 059aabb2ec5bd7b20ed08ab9c439531a352ba3ec.
I've included a patch for the relevant code included for the review. Suggestions appreciated
on whether this is the appropriate branch/commit to patch against. 
> 
> Most of the implementation is contained in vm_isolation_module.cpp and vm_isolation_module.hpp
and there are some minor additions in launcher to handle setup of the environment for the
virtual machine. I use the libvirt (http://libvirt.org/) library, to manage the virtual machine
container in which the jobs are executed. 
> 
> Dependencies 
> The code has been tested on Ubuntu 11.04 and 11.10 and depends on libpython2.6 and libvirt0

> 
> Configuration of the virtual machine container 
> The virtual machine invocation depends upon a few configuration assumptions: 
>  1. ssh public keys installed on the container. I assume that the container is setup
to allow password-less secure access. 
>   2. Directory structure on the container matches the servant machine. For example, in
invoking a spark executor, assume that the paths match the setup on the container host. 
> 
> Running it 
> In the $MESOS_HOME/conf/mesos.conf file add the line 
>    isolation=vm 
> to use the virtual machine isolation. 
> 
> The Mesos slave is invoked with the isolation parameter set to vm. For example 
>      sudo bin/mesos-slave -m mesos://master@mesos-host:5050 -w 9839 --isolation=vm 
> 
> Rough description of how it works 
> 
> The `vm_isolation_module` class forks a process that in turn launches a virtual machine.
A routine located in bin called find_addr.pl is responsible for figuring out the IP address
of the launched virtual machine. This is probably not portable since it is explicitly looking
for entry in the virbr0 network. 
> 
> A script vmLauncherTemplate.sh located in bin assists the the vmLauncher method to setup
the environment for launching tasks inside of the virtual machine. The vmLauncher method uses
vmLauncherTemplate.sh to create a tasks specific shell vmLauncherTemplate-<task_id>.sh,
which is copied to the running guest and used to run the executor inside the VM. This communicates
with the slave on the host. 
> 
> Comments and suggestions on improvements and next directions are appreciated!
> 
> 
> Diffs
> -----
> 
>   bin/find_addr.pl PRE-CREATION 
>   bin/killtree.sh PRE-CREATION 
>   bin/vmLauncher.sh PRE-CREATION 
>   bin/vmLauncherTemplate.sh PRE-CREATION 
>   src/config/config.hpp PRE-CREATION 
>   src/launcher/launcher.hpp b99b6d2 
>   src/launcher/launcher.cpp 4422224 
>   src/launcher/vm_mesos_launcher.cpp PRE-CREATION 
>   src/slave/isolation_module.cpp 5b7b4a2 
>   src/slave/isolation_module_factory.cpp 6498945 
>   src/slave/lxc_isolation_module.cpp ab0843a 
>   src/slave/main.cpp 9519ed2 
>   src/slave/slave.cpp 21fc9f2 
>   src/slave/vm_isolation_module.hpp PRE-CREATION 
>   src/slave/vm_isolation_module.cpp PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/4717/diff
> 
> 
> Testing
> -------
> 
> This was run with the spark example on single KVM virtual machine. Not tested extensively.
> 
> 
> Thanks,
> 
> Charles
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message