incubator-mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Charles Earl" <charles.ce...@gmail.com>
Subject Review Request: Basic virtual machine isolation module for Mesos
Date Sat, 14 Apr 2012 03:57:12 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4717/
-----------------------------------------------------------

Review request for mesos, Benjamin Hindman and Matei Zaharia.


Summary
-------

Earlier in the year I implemented a virtual machine isolation module. This module uses lib-virt
to launch and manage virtual machine containers. The code is still rough and have done basic
testing with the Spark example. 

This code works with the KVM (http://www.linux-kvm.org/page/Main_Page) virtual machine manager.
I've placed the relevant code in a branch called mesos-vm, for now located at https://github.com/charlescearl/VirtualMesos.
The code is based upon the mesos lxc isolation module that is located in src/slave/lxc_isolation_module.cpp/.hpp.
My code based on the mesos master branch dated Wed Nov 23 12:02:07 2011 -0800, commit 059aabb2ec5bd7b20ed08ab9c439531a352ba3ec.
I've included a patch for the relevant code included for the review. Suggestions appreciated
on whether this is the appropriate branch/commit to patch against. 

Most of the implementation is contained in vm_isolation_module.cpp and vm_isolation_module.hpp
and there are some minor additions in launcher to handle setup of the environment for the
virtual machine. I use the libvirt (http://libvirt.org/) library, to manage the virtual machine
container in which the jobs are executed. 

Dependencies 
The code has been tested on Ubuntu 11.04 and 11.10 and depends on libpython2.6 and libvirt0


Configuration of the virtual machine container 
The virtual machine invocation depends upon a few configuration assumptions: 
 1. ssh public keys installed on the container. I assume that the container is setup to allow
password-less secure access. 
  2. Directory structure on the container matches the servant machine. For example, in invoking
a spark executor, assume that the paths match the setup on the container host. 

Running it 
In the $MESOS_HOME/conf/mesos.conf file add the line 
   isolation=vm 
to use the virtual machine isolation. 

The Mesos slave is invoked with the isolation parameter set to vm. For example 
     sudo bin/mesos-slave -m mesos://master@mesos-host:5050 -w 9839 --isolation=vm 

Rough description of how it works 

The `vm_isolation_module` class forks a process that in turn launches a virtual machine. A
routine located in bin called find_addr.pl is responsible for figuring out the IP address
of the launched virtual machine. This is probably not portable since it is explicitly looking
for entry in the virbr0 network. 

A script vmLauncherTemplate.sh located in bin assists the the vmLauncher method to setup the
environment for launching tasks inside of the virtual machine. The vmLauncher method uses
vmLauncherTemplate.sh to create a tasks specific shell vmLauncherTemplate-<task_id>.sh,
which is copied to the running guest and used to run the executor inside the VM. This communicates
with the slave on the host. 

Comments and suggestions on improvements and next directions are appreciated!


Diffs
-----

  bin/find_addr.pl PRE-CREATION 
  bin/killtree.sh PRE-CREATION 
  bin/vmLauncher.sh PRE-CREATION 
  bin/vmLauncherTemplate.sh PRE-CREATION 
  src/config/config.hpp PRE-CREATION 
  src/launcher/launcher.hpp b99b6d2 
  src/launcher/launcher.cpp 4422224 
  src/launcher/vm_mesos_launcher.cpp PRE-CREATION 
  src/slave/isolation_module.cpp 5b7b4a2 
  src/slave/isolation_module_factory.cpp 6498945 
  src/slave/lxc_isolation_module.cpp ab0843a 
  src/slave/main.cpp 9519ed2 
  src/slave/slave.cpp 21fc9f2 
  src/slave/vm_isolation_module.hpp PRE-CREATION 
  src/slave/vm_isolation_module.cpp PRE-CREATION 

Diff: https://reviews.apache.org/r/4717/diff


Testing
-------

This was run with the spark example on single KVM virtual machine. Not tested extensively.


Thanks,

Charles


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message