cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donal Lafferty <>
Subject Update on Hyper-V plugin work
Date Fri, 24 May 2013 02:22:41 GMT
Reproduced from

Apologies for the length - skip to the heading that interests you.

h1. Update on Hyper-V plugin work (2013-05-23):

h2. New plugin Architecture:

h3. How does is the new architecture an improvement?

The plugin's architecture uses JSON over HTTP.  This allows the plugin agent to be written
in C# rather than Java.  .NET has tools to create WMI wrappers that allows the Hyper-V API
calls to be quickly rewritten in C#.  The result is easier to maintain and does not suffer
the IP problems of the Python used in phase 1.  An all C# agent can be contained in a single
C#/.NET Windows Service.  The result is a more robust design, because there are far fewer
points of failure.

Switching from CloudStack Message bus to JSON over HTTP communications allows the plugin's
remote agent to be implemented in C#.  Previously, this agent was written in Java to take
advantage to the existing CloudStack message bus agent.  The CloudStack message bus is difficult
to port due it's reliance on the Java NIO API. The NIO API is peculiar to Java.  As a consequence,
porting to even similar languages such as C# is difficult.  Design work is required that would
consume significant time. In contrast, JSON over HTTP passes kernel commands as JSON in the
body of an HTTP POST.  This solution is RPC-like and not RESTful, because the destination
URI that corresponds to the command and not the resource being manipulated.  The HTTP stack
removes the dependency on NIO.  HTTP stacks themselves are easy to come by regardless of the
language or platform.  Using an RPC approach avoids undertaking design work to translate incoming
kernel commands to a suitable RESTful equivalent.  Therefore, JSON over HTTP minimises the
effort required to adopt the best language for a platform.

C# code to call the Hyper-V's API directly removes IP restrictions and complexity of the existing
Python code.  Hyper-V is controlled through Windows Management Interface (WMI).  WMI provides
a uniform framework through which services and operating subsystems can provide information
and register control APIs.  All Hyper-V control packages ultimately call this API. Phase 1
of the plugin controlled Hyper-V using a port of's OpenStack driver for Hyper-V.
 The driver could be easily repurposed, because the OpenStack information model is similar
to that of CloudStack.  However, this Python code has IP, support, and coding drawbacks. 
The code is a derivative of an existing work, which prevented immediate donation to ASF. The
code used a third party WMI wrapper package that limited the version of Python that could
be used.  Finally, the python wrapper did not address discover of WMI objects, properties
and methods.  In contrast, the C# code can be transferred to ASF immediately.  The strongly
typed wrappers rely on C# and .NET, which is well supported. These wrappers are simple to
update as they are machine generated code.  More importantly, they enable property and method
discovery through autocomplete.  Therefore, the C# version is easier to maintain and addresses
Apache's copyright requirements.

Access to native error handling, APIs and services make the C#/.NET agent more robust. Previously,
the remote agent created a new process every time Python was invoked.  Communications with
this new process were over stdin and stdout.  The serialisation requirements meant exceptions
could not be used for error handling.  Also, the design was vulnerable to deadlock.  Should
the new process not return but not die either, the call would be stuck.  Now that the agent
is written entirely in C# agent, it can pass errors in a single exception rather than a series
of method returns.  Furthermore, using Python along side Java in phase 1 provides access to
platform-specific APIs not exposed by the Java Runtime Environment (JRE).  In contrast, .NET
allows direct access to Windows APIs, which means the C# agent has no need for external processes.
 Finally, additional scripts were required to start/stop/restart the phase 1 agent.  With
phase 2, the agent is implemented with hooks that allow it to be launched as a Windows Service.
 Using these hooks, the Windows Service Control Manager (SCM) takes care of executing the
agent as if it were a daemon.  Therefore, switching to a single language and process means
the new remote agent has fewer points of failure.

An additional strength of this design is that pushes the hypervisor-specific code out of the
server component.  Previously, a hypervisor's remote agent would have to register itself with
the management server.  This meant that the agent had to somehow know its zone, pod, and cluster
location before it joined itself to the cloud.  For example, the KVM server component uses
SSH to launch the remote agent with the necessary configuration data.  In our new design,
the remote agent can be told its cloud configuration after it has been started.  This removes
the need for a hypervisor-specific control to launch remote agent.  Therefore, the resulting
server component is much more reusable.

To summarise, the plugin has an updated communications stack called JSON over HTTP.  This
has simplified the implementation of the remote agent and the corresponding server component.
 The result is a more robust and reusable design, and a new implementation that meets Apache
CloudStack's IP requirements.

h2. Implementation Status:

A VisualStudio 2012 SP1 solution on GitHub ([])
holds the plugin's new remote agent.  VM and volume creation / deletion commands are mostly
complete. Additional storage commands are required to allow transfer of template and volume
images.  Attach / detach commands should be added to allow System VM patching.

The solution includes unit tests.  The tests are a port of the functional tests from phase
1.  At the time of writing, the port was still being completed.

The implementation avoids using features that would require the hypervisor to run Windows
Server 2012 in the parent partition.  The image store uses S3 for template storage, because
the Windows NFS client is an advanced feature that is not available with the basic and free
Hyper-V Server 2012.  Likewise, the remote agent hosts its own HTTP stack to avoid any dependency
of Microsoft's Internet Information Services (IIS) web server.

A Server component is available that uses JSON over HTTP to communicate the remote agent.
 The existing unit tests are being updated to call this server component.  Additional scripting
is required to launch the remote agent so that HTTP requests can be properly received and
acted on.  The server component's Discoverer algorithm needs to be updated.

The build and test system has dependencies on Microsoft tools and platforms.  To build the
system with free tools, the remote agent should be compiled against VisualStudio express.
 To build with Linux, the solution should be compiled with Mono.  However, neither of these
options have been tried.

h2. IP Dependencies

The C# source code and configuration files that are input for code generate are in the process
of being marked with the Apache header ([])
to indicate that Apache CloudStack will hold the copyright.

Third party binaries used by the solution are the Microsoft .NET Framework 4.5, NuGet packages,
and AWS .NET SDK.  The AWS SDK is used to access S3 storage.  The NuGet packages provide logging,
JSON serialisation, and a light weight HTTP stack.  The background to NuGet is that it is
akin to a Maven repo, but each packages has a corresponding web page that includes details
of the license for use of the package.

Licenses for the NuGet packages are enumerated below.  I do not have the .NET Framework 4.5
EULA to hand.

Apache License, Version 2.0 ([]:

Log4net 2.0.0 ([])

The MIT License (MIT):

Newtonsoft.Json 4.5.11 []


Microsoft ASP.NET Web API Core Libraries 4.0.20710.0 []
NuGet id="Microsoft.AspNet.WebApi.Core"  version="4.0.20710.0"
Microsoft ASP.NET Web API Client Libraries 4.0.20710.0 []
NuGet id="Microsoft.AspNet.WebApi.Client" version="4.0.20710.0"
Microsoft .NET Framework 4 HTTP Client Libraries 2.0.20710.0 []
NuGet id="Microsoft.Net.Http" version="2.0.20710.0"
Microsoft ASP.NET Web API Self Host 4.0.20918.0 []
NuGet id="Microsoft.AspNet.WebApi.SelfHost" version="4.0.20918.0"

h2. System VM services:

The plugin depends on system VM services that are coded separately.  System VM services refer
to CloudStack functionality that is offloaded from the management server to VMs that sit in
the cloud.  The services in question are template/snapshot storage, networking services, and
console access.  Traditionally, CloudStack has implemented these services using system VMs:
 secondary storage VM for image persistence, virtual router VM for networking services, and
console VM for access to guest VM consoles.  For the Hyper-V plugin to work properly, CloudStack
has to be able to offer these same services on Hyper-V hypervisors.

The image that the CloudStack database points to for Hyper-V system VM needs to be updated
with a version that includes the console proxy.  The latest system VM includes Hyper-V kernel
modules.  This provides it with the high performance network driver to act as a virtual router
VM.  The secondary storage VM capabilities are not required.  The secondary storage VM allows
CloudStack to upload images to an NFS store.  However, a NFS client is not available in the
"Hyper-V Server" that we are targeting.  Therefore, we will not be using NFS for image storage.
 Finally, the console services must be updated.  Hyper-V's guest consoles are accessed through
RDP, which means an RDP proxy must be added to the system VM.  Therefore, changes to the system
VM appear to be limited to the addition of a Hyper-V compatible console proxy.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message