cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donal Lafferty <>
Subject RE: Update on Hyper-V plugin work
Date Mon, 27 May 2013 16:38:23 GMT
 Hi Chip,

Thanks for clarifying the IP info required.  I'll take a closer look ASAP.

WRT collaboration, on the agent-side, I did the design work, and used colleagues working on
CloudStack features for learning and some discussion.  Reviews requests have flowed through
this mailing list.  However, the final architecture meant to ease scaling out development.
 The stateless agent and HTTP request model should allow developers to add commands without
stepping on each other's toes.
On the System VM side, the community updated the storage architecture such that CloudStack
4.2 will expect Hyper-V to use S3, and they updated the system VM such that it is using a
kernel with Hyper-V PV drivers.  Citrix have provided some effort for a suitable console proxy,
which needs to be integrated into the System VM.  (Volunteers?)


-----Original Message-----
From: Chip Childers [] 
Sent: 24 May 2013 1:28 PM
Subject: Re: Update on Hyper-V plugin work


First, thank you for sending this out!  It seems like really good progress.  Who are you working
with on this?  Are you running solo?

The technical elements of this seem sound at first review.

The licensing implications are going to take some time to parse through though.  One of the
things that could help us on that front, would be to understand which of the licenses are
runtime vs. buildtime.  It would also be good to highlight which dependencies would easily
be considered "system dependencies" (as in: expected to be installed prior to our software
being installed).

Again, thanks!


On Fri, May 24, 2013 at 02:22:41AM +0000, Donal Lafferty wrote:
> Reproduced from 
> Apologies for the length - skip to the heading that interests you.
> h1. Update on Hyper-V plugin work (2013-05-23):
> h2. New plugin Architecture:
> e2_HyperV_Remote_Agent.png
> h3. How does is the new architecture an improvement?
> The plugin's architecture uses JSON over HTTP.  This allows the plugin agent to be written
in C# rather than Java.  .NET has tools to create WMI wrappers that allows the Hyper-V API
calls to be quickly rewritten in C#.  The result is easier to maintain and does not suffer
the IP problems of the Python used in phase 1.  An all C# agent can be contained in a single
C#/.NET Windows Service.  The result is a more robust design, because there are far fewer
points of failure.
> Switching from CloudStack Message bus to JSON over HTTP communications allows the plugin's
remote agent to be implemented in C#.  Previously, this agent was written in Java to take
advantage to the existing CloudStack message bus agent.  The CloudStack message bus is difficult
to port due it's reliance on the Java NIO API. The NIO API is peculiar to Java.  As a consequence,
porting to even similar languages such as C# is difficult.  Design work is required that would
consume significant time. In contrast, JSON over HTTP passes kernel commands as JSON in the
body of an HTTP POST.  This solution is RPC-like and not RESTful, because the destination
URI that corresponds to the command and not the resource being manipulated.  The HTTP stack
removes the dependency on NIO.  HTTP stacks themselves are easy to come by regardless of the
language or platform.  Using an RPC approach avoids undertaking design work to translate incoming
kernel commands to a suitable RESTful equivalent.  Therefore, JSON over HTTP minimises the
effort required to adopt the best language for a platform.
> C# code to call the Hyper-V's API directly removes IP restrictions and complexity of
the existing Python code.  Hyper-V is controlled through Windows Management Interface (WMI).
 WMI provides a uniform framework through which services and operating subsystems can provide
information and register control APIs.  All Hyper-V control packages ultimately call this
API. Phase 1 of the plugin controlled Hyper-V using a port of's OpenStack driver
for Hyper-V.  The driver could be easily repurposed, because the OpenStack information model
is similar to that of CloudStack.  However, this Python code has IP, support, and coding drawbacks.
 The code is a derivative of an existing work, which prevented immediate donation to ASF.
The code used a third party WMI wrapper package that limited the version of Python that could
be used.  Finally, the python wrapper did not address discover of WMI objects, properties
and methods.  In contrast, the C# code can be transferred to ASF immediately.  The strongly
typed wrappers rely on C# and .NET, which is well supported. These wrappers are simple to
update as they are machine generated code.  More importantly, they enable property and method
discovery through autocomplete.  Therefore, the C# version is easier to maintain and addresses
Apache's copyright requirements.
> Access to native error handling, APIs and services make the C#/.NET agent more robust.
Previously, the remote agent created a new process every time Python was invoked.  Communications
with this new process were over stdin and stdout.  The serialisation requirements meant exceptions
could not be used for error handling.  Also, the design was vulnerable to deadlock.  Should
the new process not return but not die either, the call would be stuck.  Now that the agent
is written entirely in C# agent, it can pass errors in a single exception rather than a series
of method returns.  Furthermore, using Python along side Java in phase 1 provides access to
platform-specific APIs not exposed by the Java Runtime Environment (JRE).  In contrast, .NET
allows direct access to Windows APIs, which means the C# agent has no need for external processes.
 Finally, additional scripts were required to start/stop/restart the phase 1 agent.  With
phase 2, the agent is implemented with hooks that allow it to be launched as a Windows Service.
 Using these hooks, the Windows Service Control Manager (SCM) takes care of executing the
agent as if it were a daemon.  Therefore, switching to a single language and process means
the new remote agent has fewer points of failure.
> An additional strength of this design is that pushes the hypervisor-specific code out
of the server component.  Previously, a hypervisor's remote agent would have to register itself
with the management server.  This meant that the agent had to somehow know its zone, pod,
and cluster location before it joined itself to the cloud.  For example, the KVM server component
uses SSH to launch the remote agent with the necessary configuration data.  In our new design,
the remote agent can be told its cloud configuration after it has been started.  This removes
the need for a hypervisor-specific control to launch remote agent.  Therefore, the resulting
server component is much more reusable.
> To summarise, the plugin has an updated communications stack called JSON over HTTP. 
This has simplified the implementation of the remote agent and the corresponding server component.
 The result is a more robust and reusable design, and a new implementation that meets Apache
CloudStack's IP requirements.
> h2. Implementation Status:
> A VisualStudio 2012 SP1 solution on GitHub ([])
holds the plugin's new remote agent.  VM and volume creation / deletion commands are mostly
complete. Additional storage commands are required to allow transfer of template and volume
images.  Attach / detach commands should be added to allow System VM patching.
> The solution includes unit tests.  The tests are a port of the functional tests from
phase 1.  At the time of writing, the port was still being completed.
> The implementation avoids using features that would require the hypervisor to run Windows
Server 2012 in the parent partition.  The image store uses S3 for template storage, because
the Windows NFS client is an advanced feature that is not available with the basic and free
Hyper-V Server 2012.  Likewise, the remote agent hosts its own HTTP stack to avoid any dependency
of Microsoft's Internet Information Services (IIS) web server.
> A Server component is available that uses JSON over HTTP to communicate the remote agent.
 The existing unit tests are being updated to call this server component.  Additional scripting
is required to launch the remote agent so that HTTP requests can be properly received and
acted on.  The server component's Discoverer algorithm needs to be updated.
> The build and test system has dependencies on Microsoft tools and platforms.  To build
the system with free tools, the remote agent should be compiled against VisualStudio express.
 To build with Linux, the solution should be compiled with Mono.  However, neither of these
options have been tried.
> h2. IP Dependencies
> The C# source code and configuration files that are input for code generate are in the
process of being marked with the Apache header ([])
to indicate that Apache CloudStack will hold the copyright.
> Third party binaries used by the solution are the Microsoft .NET Framework 4.5, NuGet
packages, and AWS .NET SDK.  The AWS SDK is used to access S3 storage.  The NuGet packages
provide logging, JSON serialisation, and a light weight HTTP stack.  The background to NuGet
is that it is akin to a Maven repo, but each packages has a corresponding web page that includes
details of the license for use of the package.
> Licenses for the NuGet packages are enumerated below.  I do not have the .NET Framework
4.5 EULA to hand.
> Apache License, Version 2.0 ([]:
> AWS .NET SDK ([])
> Log4net 2.0.0 ([])
> The MIT License (MIT):
> Newtonsoft.Json 4.5.11 []
> ([])
> Microsoft ASP.NET Web API Core Libraries 4.0.20710.0 []
NuGet id="Microsoft.AspNet.WebApi.Core"  version="4.0.20710.0"
> Microsoft ASP.NET Web API Client Libraries 4.0.20710.0 []
NuGet id="Microsoft.AspNet.WebApi.Client" version="4.0.20710.0"
> Microsoft .NET Framework 4 HTTP Client Libraries 2.0.20710.0 []
NuGet id="Microsoft.Net.Http" version="2.0.20710.0"
> Microsoft ASP.NET Web API Self Host 4.0.20918.0 []
NuGet id="Microsoft.AspNet.WebApi.SelfHost" version="4.0.20918.0"
> h2. System VM services:
> The plugin depends on system VM services that are coded separately.  System VM services
refer to CloudStack functionality that is offloaded from the management server to VMs that
sit in the cloud.  The services in question are template/snapshot storage, networking services,
and console access.  Traditionally, CloudStack has implemented these services using system
VMs:  secondary storage VM for image persistence, virtual router VM for networking services,
and console VM for access to guest VM consoles.  For the Hyper-V plugin to work properly,
CloudStack has to be able to offer these same services on Hyper-V hypervisors.
> The image that the CloudStack database points to for Hyper-V system VM needs to be updated
with a version that includes the console proxy.  The latest system VM includes Hyper-V kernel
modules.  This provides it with the high performance network driver to act as a virtual router
VM.  The secondary storage VM capabilities are not required.  The secondary storage VM allows
CloudStack to upload images to an NFS store.  However, a NFS client is not available in the
"Hyper-V Server" that we are targeting.  Therefore, we will not be using NFS for image storage.
 Finally, the console services must be updated.  Hyper-V's guest consoles are accessed through
RDP, which means an RDP proxy must be added to the system VM.  Therefore, changes to the system
VM appear to be limited to the addition of a Hyper-V compatible console proxy.

View raw message