cloudstack-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From seb...@apache.org
Subject git commit: updated refs/heads/master to fd32c33
Date Mon, 29 Jul 2013 12:30:32 GMT
Updated Branches:
  refs/heads/master 62d84d119 -> fd32c33d6


GSOC report for SDN project

Signed-off-by: Sebastien Goasguen <runseb@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/cloudstack/repo
Commit: http://git-wip-us.apache.org/repos/asf/cloudstack/commit/fd32c33d
Tree: http://git-wip-us.apache.org/repos/asf/cloudstack/tree/fd32c33d
Diff: http://git-wip-us.apache.org/repos/asf/cloudstack/diff/fd32c33d

Branch: refs/heads/master
Commit: fd32c33d6d278ec5913bb62b6828adb945036a28
Parents: 62d84d1
Author: tuna <ng.tuna@gmail.com>
Authored: Mon Jul 29 08:30:17 2013 -0400
Committer: Sebastien Goasguen <runseb@gmail.com>
Committed: Mon Jul 29 08:30:17 2013 -0400

----------------------------------------------------------------------
 docs/en-US/gsoc-midsummer-nguyen.xml  | 456 ++++++++++++++++++++++++++++-
 docs/en-US/images/l3_services.png     | Bin 0 -> 30362 bytes
 docs/en-US/images/network_service.png | Bin 0 -> 28162 bytes
 docs/en-US/images/odl_structure.jpg   | Bin 0 -> 116542 bytes
 4 files changed, 454 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/cloudstack/blob/fd32c33d/docs/en-US/gsoc-midsummer-nguyen.xml
----------------------------------------------------------------------
diff --git a/docs/en-US/gsoc-midsummer-nguyen.xml b/docs/en-US/gsoc-midsummer-nguyen.xml
index 96c2de1..b4f4f5a 100644
--- a/docs/en-US/gsoc-midsummer-nguyen.xml
+++ b/docs/en-US/gsoc-midsummer-nguyen.xml
@@ -23,6 +23,458 @@
 -->
 
 <section id="gsoc-midsummer-nguyen">
-    <title>Mid-Summer Progress Updates</title>
-    <para>This section describes ...</para>
+    <title>Mid-Summer Progress Updates for Nguyen Anh Tu - "Add Xen/XCP support for
GRE SDN controller"</title>
+    <para>This section describes my progress with the project titled "Add Xen/XCP support
for GRE SDN controller"</para>
+    <section id="Introduction">
+	<title>Introduction</title>
+	<para>It has been a half way of GSoC2013 journey which I am getting more familiar
with its activities. Personally, the previous one-and-a-half month has surprisingly passed
by in a blink with lots of pressure. In this first time joining in GSoC2013, I have found
it totally new and interesting in its working methods and challenges. Along with those stressful
moments, I appreciated all wonderful experiences and knowledge that I have luckily gained
from this commitment. It is time to review it all and present in time order.
+	</para>
+	<para>My project named “Add Xen/XCP support for GRE SDN controller”, the proposal
can be found here: <ulink url="http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/ngtuna/1">Proposal</ulink>
+	</para>
+	<para>Specifically, I need to improve the current GRE SDN controller to work with
XCP, a free version of XenServer. Then, as mentioning with my two mentor Sebastien Goasguen
and Hugo, I continue to work in next missions as below:
+	</para>
+	<itemizedlist>
+		<listitem><para>re-factor GRE source code by following NiciraNVP plugin design.</para></listitem>
+		<listitem><para>add GRE support for KVM hypervisor.</para></listitem>
+		<listitem><para>develop a new ODL plugin using Opendaylight controller for
controlling and managing network services via OpenFlow protocol.</para></listitem>
+	</itemizedlist>
+	<para>At the beginning, I started to explore frameworks and tools that CloudStack
uses such as Spring framework, marven, git and Reviewboard. In my country developers are more
familiar with svn than git, however these tools are also such easy to use so I don't write
more about them. I want to note about using Spring in CloudStack and what happen in the Management
Server startup process.
+	</para>
+    </section>
+    <section id="Spring">
+	<title>Spring in CloudStack</title>
+	<para>Spring provides a Container which contains pre-loaded components CloudStack
use. At startup, these components are loaded to Container via two ways:
+	</para>
+	<itemizedlist>
+		<listitem>
+		<para>components are declared as beans in componentcontext.xml and applicationcontext.xml</para>
+		<programlisting language="Java">
+			&lt;bean id="accountDaoImpl" class="com.cloud.user.dao.AccountDaoImpl" /&gt;
+			&lt;bean id="accountDetailsDaoImpl" class="com.cloud.user.AccountDetailsDaoImpl" /&gt;
+			&lt;bean id="accountJoinDaoImpl" class="com.cloud.api.query.dao.AccountJoinDaoImpl"
/&gt;
+			&lt;bean id="accountGuestVlanMapDaoImpl" class="com.cloud.network.dao.AccountGuestVlanMapDaoImpl"
/&gt;
+			&lt;bean id="accountVlanMapDaoImpl" class="com.cloud.dc.dao.AccountVlanMapDaoImpl"
/&gt;
+			...
+		</programlisting>
+		</listitem>
+		<listitem>
+		<para>components are marked with @Component annotation</para>
+		<programlisting language="Java">
+			@Component
+			@Local(value = { NetworkManager.class})
+			public class NetworkManagerImpl extends ManagerBase implements NetworkManager, Listener
{
+			    static final Logger s_logger = Logger.getLogger(NetworkManagerImpl.class);
+		</programlisting>
+		</listitem>
+	</itemizedlist>
+	<para>As I know recently @Component is not recommended.</para>
+	<para>The fundamental functionality provided by the Spring Container is Dependency
Injection (DI). To decouple Java components from other Java components the dependency to a
certain other class should get injected into them rather that the class inself creates or
finds this object. The general concept between dependency injection is called Inversion of
Control. A class should not configure itself but should be configured from outside. A design
based on independent classes / components increases the re-usability and possibility to test
the software. Example of using DI in CloudStack is showed below:</para>
+	<programlisting language="Java">
+	public class NetworkManagerImpl extends ManagerBase implements NetworkManager, Listener
{
+	    static final Logger s_logger = Logger.getLogger(NetworkManagerImpl.class);
+
+	    @Inject
+	    DataCenterDao _dcDao = null;
+	    @Inject
+	    VlanDao _vlanDao = null;
+	    @Inject
+	    IPAddressDao _ipAddressDao = null;
+	    @Inject
+	    AccountDao _accountDao = null;
+	</programlisting>
+    </section>
+    <section id="MsStartup">
+	<title>Management Server Startup</title>
+	<para>The MS startup process is defined in cloud-client-ui/WEB-INF/web.xml. The following
items will be loaded sequentially:</para>
+	<itemizedlist>
+		<listitem><para>Log4jConfigListener.</para></listitem>
+		<listitem><para>ContextLoaderListener.</para></listitem>
+		<listitem><para>CloudStartupServlet.</para></listitem>
+		<listitem><para>ConsoleServlet.</para></listitem>
+		<listitem><para>ApiServlet.</para></listitem>
+	</itemizedlist>
+	<para>Of which, CloudStartupServlet will call to ComponentContext to init all of pre-defined
components life cycle including configure() and start() phase. The components are divided
into seven levels to consecutively startup. Of course, they must override configure() and
start() methods.</para>
+	<programlisting language="Java">
+	public interface ComponentLifecycle {
+		public static final int RUN_LEVEL_SYSTEM_BOOTSTRAP = 0;		// for system level bootstrap
components
+		public static final int RUN_LEVEL_SYSTEM = 1;				// for system level service components
(i.e., DAOs)
+		public static final int RUN_LEVEL_FRAMEWORK_BOOTSTRAP = 2;	// for framework startup checkers
(i.e., DB migration check)
+		public static final int RUN_LEVEL_FRAMEWORK = 3;			// for framework bootstrap components(i.e.,
clustering management components)
+		public static final int RUN_LEVEL_COMPONENT_BOOTSTRAP = 4;	// general manager components
+		public static final int RUN_LEVEL_COMPONENT = 5;			// regular adapters, plugin components
+		public static final int RUN_LEVEL_APPLICATION_MAINLOOP = 6;
+		public static final int MAX_RUN_LEVELS = 7;
+	</programlisting>
+	<programlisting language="Java">
+	    	// configuration phase
+		Map&lt;String, String> avoidMap = new HashMap&lt;String, String>();
+	    	for(int i = 0; i &lt; ComponentLifecycle.MAX_RUN_LEVELS; i++) {
+	    		for(Map.Entry&lt;String, ComponentLifecycle> entry : ((Map&lt;String, ComponentLifecycle>)classifiedComponents[i]).entrySet())
{
+	    			ComponentLifecycle component = entry.getValue();
+	    			String implClassName = ComponentContext.getTargetClass(component).getName();
+		        s_logger.info("Configuring " + implClassName);
+		        
+		        if(avoidMap.containsKey(implClassName)) {
+		            s_logger.info("Skip configuration of " + implClassName + " as it is already
configured");
+		        	continue;
+		        }
+		        
+		        try {
+						component.configure(component.getName(), component.getConfigParams());
+					} catch (ConfigurationException e) {
+						s_logger.error("Unhandled exception", e);
+						throw new RuntimeException("Unable to configure " + implClassName, e);
+					}
+		        
+		        avoidMap.put(implClassName, implClassName);
+	    		}
+	    	}
+	</programlisting>
+	<programlisting language="Java">
+	    	// starting phase
+	    	avoidMap.clear();
+	    	for(int i = 0; i &lt; ComponentLifecycle.MAX_RUN_LEVELS; i++) {
+	    		for(Map.Entry&lt;String, ComponentLifecycle> entry : ((Map&lt;String, ComponentLifecycle>)classifiedComponents[i]).entrySet())
{
+	    			ComponentLifecycle component = entry.getValue();
+	    			String implClassName = ComponentContext.getTargetClass(component).getName();
+		        s_logger.info("Starting " + implClassName);
+		        
+		        if(avoidMap.containsKey(implClassName)) {
+		            s_logger.info("Skip configuration of " + implClassName + " as it is already
configured");
+		        	continue;
+		        }
+		        
+		        try {
+						component.start();
+					
+						if(getTargetObject(component) instanceof ManagementBean)
+							registerMBean((ManagementBean)getTargetObject(component));
+					} catch (Exception e) {
+						s_logger.error("Unhandled exception", e);
+						throw new RuntimeException("Unable to start " + implClassName, e);
+					}
+		        
+		        avoidMap.put(implClassName, implClassName);
+	    		}
+	    	}
+	</programlisting>
+    </section>
+    <section id="networking">
+	<title>Network Architecture</title>
+	<para>Networking is the most important component in CloudStack, which serves network
services from layer 2 to layer 7. In GsoC, fortunately I have a chance to learn about CloudsStack
network architecture. It's really amazing. CloudStack's networking is divided to three parts:</para>
+	<para><emphasis role="bold">NetworkGuru</emphasis></para>
+	<para>NetworkGuru are responsible for:</para>
+	<itemizedlist>
+		<listitem><para>Design and implementation of virtual networks.</para></listitem>
+		<listitem><para>IP adress management.</para></listitem>
+	</itemizedlist>
+	<para>See full description about Network Guru on my wiki post: <ulink url="https://cwiki.apache.org/confluence/display/CLOUDSTACK/Add+Xen+and+XCP+support+for+GRE+SDN+controller">Add
Xen/XCP support for GRE SDN controller</ulink></para>
+	<para><emphasis role="bold">NetworkElement</emphasis></para>
+	<para>NetworkElement in my opinion is the most important in CloudStack's networking.
It represents components that are present in network. Such components can provide any kind
of network service or support the virtual networking infrastructure and their interface is
defined by com.cloud.network.element.NetworkElement. There are two things we attend in NetworkElement:
services and elements.</para>
+	<para>CloudStack currently support network services below:</para>
+	<itemizedlist>
+		<listitem><para>Dhcp service.</para></listitem>
+		<listitem><para>Connectivity service.</para></listitem>
+		<listitem><para>Firewall service.</para></listitem>
+		<listitem><para>Load Balancing service.</para></listitem>
+		<listitem><para>Network ACL service.</para></listitem>
+		<listitem><para>Port Forwarding service.</para></listitem>
+		<listitem><para>SourceNat service.</para></listitem>
+		<listitem><para>StaticNat service.</para></listitem>
+		<listitem><para>UerData service.</para></listitem>
+		<listitem><para>Vpc service.</para></listitem>
+	</itemizedlist>
+	<para>Many Element implemented these above services. They are:</para>
+	<itemizedlist>
+		<listitem><para>MidonetElement.</para></listitem>
+		<listitem><para>BigSwitchVnsElement.</para></listitem>
+		<listitem><para>NiciraNvpElement.</para></listitem>
+		<listitem><para>BaremetalElement.</para></listitem>
+		<listitem><para>VirtualRouterElement.</para></listitem>
+		<listitem><para>VpcVirtualRouterElement.</para></listitem>
+		<listitem><para>CiscoVnmcElement.</para></listitem>
+		<listitem><para>JuniperSrxExternalFirewallElement.</para></listitem>
+		<listitem><para>ElasticLbElement.</para></listitem>
+		<listitem><para>F5ExternalLbElement.</para></listitem>
+		<listitem><para>CloudZoneNetworkElement.</para></listitem>
+		<listitem><para>BaremetalPxeElement.</para></listitem>
+		<listitem><para>BaremetalUserdataElement.</para></listitem>
+		<listitem><para>DnsNotifier.</para></listitem>
+		<listitem><para>OvsElement.</para></listitem>
+		<listitem><para>SecurityGroupElement.</para></listitem>
+	</itemizedlist>
+	<para>See full description about Network Element on my wiki post: <ulink url="https://cwiki.apache.org/confluence/display/CLOUDSTACK/Add+Xen+and+XCP+support+for+GRE+SDN+controller">Add
Xen/XCP support for GRE SDN controller</ulink></para>
+	<para>In addition, Elements willing to support network services have to implement
corresponding methods from ServicesProvider interfaces. For example, NiciraNvpElement want
to support staticNat rule so it has to override applyStaticNats method.</para>
+	<para><emphasis role="bold">NetworkManager</emphasis></para>
+	<para>Network Manager handle the resources managed by the network elements. They are
also implemented as many other "resource" managers in CloudStack.</para>
+	<para>For instance, the manager for setting up L2-in-L3 networks with Open vSwitch
is OvsTunnelManagerImpl, whereas Virtual Router lifecycle is managed by VirtualApplianceManagerImpl.</para>
+	<para>In the project, I'm going to implement L3 services for sdn controller, so I
need to understand how network services implement.</para>
+    </section>
+    <section id="networkservices">
+	<title>Network Services</title>
+	<para>As I said in previous session, network services are represented in ServiceProvider
interfaces. There are currently 12 service providers including: Dhcp, Firewall, IpDeployer,
LoadBalancing, NetworkACL, PortForwarding, RemoteAccessVpn, Site2siteVpn, SourceNat, StaticNat,
UserData and Vpc. In this session,  I'll focus on L3 services implemented in CloudStack such
as FirewallRule, PortForwardingRule, StaticNatRules, etc. All services are implemented at
NetworkElement and every elements including network plugins (nicira nvp, bigswitch vns,...),
which is willing to support them, must override from NetworkElement. For a clearly exlaination,
I'll take the StaticNat service implemented in Nicira NVP plugin, source code can be found
in NiciraNvpElement.java.</para>
+	<para>NiciraNvpElement firstly has to check whether it can handle the StaticNat service
via canHandle() method:</para>
+	<programlisting language="Java">
+		if (!canHandle(network, Service.StaticNat)) {
+		    return false;
+		}
+	</programlisting>
+	<programlisting language="Java">
+	    protected boolean canHandle(Network network, Service service) {
+		s_logger.debug("Checking if NiciraNvpElement can handle service "
+		        + service.getName() + " on network " + network.getDisplayText());
+
+		//Check if network has right broadcast domain type
+		if (network.getBroadcastDomainType() != BroadcastDomainType.Lswitch) {
+		    return false;
+		}
+
+		//Check if NiciraNVP is the provider of the network
+		if (!_networkModel.isProviderForNetwork(getProvider(),
+		        network.getId())) {
+		    s_logger.debug("NiciraNvpElement is not a provider for network "
+		            + network.getDisplayText());
+		    return false;
+		}
+
+		//Check if NiciraNVP support StaticNat service
+		if (!_ntwkSrvcDao.canProviderSupportServiceInNetwork(network.getId(),
+		        service, Network.Provider.NiciraNvp)) {
+		    s_logger.debug("NiciraNvpElement can't provide the "
+		            + service.getName() + " service on network "
+		            + network.getDisplayText());
+		    return false;
+		}
+
+		return true;
+	    }
+	</programlisting>
+	<para>NiciraNvp checks whether it is the provider of the network and it can support
StaticNat service or not. After the checking, it makes a staticNat rely on their own Logical
Router, that I won't report detail here.</para>
+	<para>The sequence diagram for applying a L3 service is described below:</para>
+	<mediaobject>
+		<imageobject>
+		    <imagedata fileref="./images/network_service.png" />
+		</imageobject>
+		<textobject><phrase>network_service.png: Network services implementation sequence
diagram.</phrase></textobject>
+	</mediaobject>
+	<para>After understanding network architecture and services implementation, I decided
to improve Ovs plugin to support L3 services. Because it's the native sdn controller, I want
to use Virtual Router for L3 services deployment. This work will be done when I call L3 services
execution from OvsElement to VirtualRouterManager. With Xen hosts, VirtualRouterElement execute
L3 services via xapi plugin calls. I make a flow which describes more detail about the process
below</para>
+	<mediaobject>
+		<imageobject>
+		    <imagedata fileref="./images/l3_services.png" />
+		</imageobject>
+		<textobject><phrase>l3_services.png: Layer 3 services implementation in Ovs
plugin.</phrase></textobject>
+	</mediaobject>
+	<para>In Xen, all of L3 services are executed via a Xapi plugin naming "vmops". Default,
Virtual Routers (VR) control and manage network services. In this case, "vmops" forwards request
to network-responsibility shellscripts such as call_firewall.sh or call_loadbalancer.sh. They
then parse parameters and call to shellscripts placed in VR via ssh. For example, if we define
a staticNat rule, the process occurs as follow:</para>
+	<para>VR Manager (VirtualNetworkApplianceManager) send staticNat command to AgentManager:</para>
+	<programlisting language="Java">
+		try {
+		    answers = _agentMgr.send(router.getHostId(), cmds);
+		} catch (OperationTimedoutException e) {
+		    s_logger.warn("Timed Out", e);
+		    throw new AgentUnavailableException("Unable to send commands to virtual router ", router.getHostId(),
e);
+		}
+	</programlisting>
+	<para>AgentManager makes a xapi plugin call to host containing the VR</para>
+	<programlisting language="Java">
+		String result = callHostPlugin(conn, "vmops", "setFirewallRule", "args", args.toString());
+	</programlisting>
+	<para>"vmops" forwards the request to "call_firewall" shellscript</para>
+	<programlisting language="Python">
+		@echo
+		def setFirewallRule(session, args):
+		    sargs = args['args']
+		    cmd = sargs.split(' ')
+		    cmd.insert(0, "/usr/lib/xcp/bin/call_firewall.sh")
+		    cmd.insert(0, "/bin/bash")
+		    try:
+			txt = util.pread2(cmd)
+			txt = 'success'
+		    except:
+			util.SMlog(" set firewall rule failed "  )
+			txt = '' 
+
+		    return txt
+	</programlisting>
+	<para>"call_firewall" parses the parameters and directly request to a shellscript
placed in VR via ssh command</para>
+	<programlisting>
+		ssh -p 3922 -q -o StrictHostKeyChecking=no -i $cert root@$domRIp "/root/firewall.sh $*"
+	</programlisting>
+	<para>That's all. "firewall" script set some iptable rules for executing the staticNat
rule</para>
+    </section>
+    <section id="opendaylight">
+	<title>Opendaylight Controller</title>
+	<para>The project need to add an open source Openflow controller, and I decided to
choose Opendaylight.</para>
+	<para>Opendaylight (ODL) is an interesting experience that I have in GSoC. Before
starting project, I still confused between many open source OpenFlow controller such as POX,
NOX, Beacon, Floodlight, Opendaylight... Honestly, I do not have large knowledge of OpenFlow
protocol and also open source SDN controller at the beginning of project. When the project
was in progress, I chose Floodlight, a safe solution because of its rich of functionality
and good documents. However, Sebastien Goasguen, CloudStack GSoC manager, recommended me to
try Opendaylight. From the collected information, I found that Opendaylight are getting a
lot of attentions from the community.</para>
+	<para>At the moment, ODL has three main projects:</para>
+	<itemizedlist>
+		<listitem><para>Opendaylight Controller.</para></listitem>
+		<listitem><para>Opendaylight Network Virtualization Platform.</para></listitem>
+		<listitem><para>Opendaylight Virtual Tennant Network.</para></listitem>
+	</itemizedlist>
+	<para>It also has six incubating projects:</para>
+	<itemizedlist>
+		<listitem><para>YANG Tools.</para></listitem>
+		<listitem><para>LISP Flow Mapping.</para></listitem>
+		<listitem><para>OVSDB Integration.</para></listitem>
+		<listitem><para>Openflow Protocol Library.</para></listitem>
+		<listitem><para>BGP-LS/PCEP.</para></listitem>
+		<listitem><para>Defense4All.</para></listitem>
+	</itemizedlist>
+	<para>For integrating Opendaylight to control and manage network services, I chose
ODL Controller project, which is developed by Cisco programmers. The ODL controller is a pure
software and as a JVM it can be run on any OS as long as it supports Java. The structure of
the ODL controller is shown below:</para>
+	<mediaobject>
+		<imageobject>
+		    <imagedata fileref="./images/odl_structure.jpg" />
+		</imageobject>
+		<textobject><phrase>odl_structure.jpg: Opendaylight Controller architecture.</phrase></textobject>
+	</mediaobject>
+	<para>The structure is separated to three layers:</para>
+	<itemizedlist>
+		<listitem><para><emphasis role="bold">Network Apps and Orchestration</emphasis>:
the top layer consists of applications that utilize the network for normal network communications.
Also included in this layer are business and network logic applications that control and monitor
network behavior.</para></listitem>
+		<listitem><para><emphasis role="bold">Controller Platform</emphasis>:
the middle layer is the framework in which the SDN abstractions can manifest; providing a
set of common APIs to the application layer (commonly referred to as the northbound interface),
while implementing one or more protocols for command and control of the physical hardware
within the network (typically referred to as the southbound interface).</para></listitem>
+		<listitem><para><emphasis role="bold">Physical and Virtual Network Devices</emphasis>:
The bottom layer consists of the physical and virtual devices, switches, routers, etc., that
make up the connective fabric between all endpoints within the network.</para></listitem>
+	</itemizedlist>
+	<para>This controller is implemented strictly in software and is contained within
its own Java Virtual Machine (JVM).</para>
+	<para>Source code can be cloned from git:</para>
+	<programlisting>
+		git clone https://git.opendaylight.org/gerrit/p/controller.git
+	</programlisting>
+	<para>Applications make request to ODL Northbound API via HTTP. Currently, ODL supports
not too much services. All REST API we can find here: <ulink url="https://wiki.opendaylight.org/view/OpenDaylight_Controller:REST_Reference_and_Authentication">ODL
Controller REST API</ulink></para>
+	<para>For example, we can add query list of exist flows configured on a Node in a
give container.</para>
+	<programlisting>
+		GET http://<emphasis>controller-ip</emphasis>/controller/nb/v2/flow/{containerName}/{nodeType}/{nodeId}
+		{containername}: name of the container. The container name for the base controller is “default”
+		{nodeType}: type of the node being programmed
+		{nodeId}: node identifier
+	</programlisting>
+	<para>Or we can add a new flow</para>
+	<programlisting>
+		POST http://<emphasis>controller-ip</emphasis>/controller/nb/v2/flow/{containerName}/{nodeType}/{nodeId}/{name}
+	</programlisting>
+	<para>with request body in XML or JSON format</para>
+	<programlisting>
+		{ "actions" : [ "...", ... ],
+		"nwDst" : "...",
+		"hardTimeout" : "...",
+		"installInHw" : "...",
+		"tosBits" : "...",
+		"cookie" : "...",
+		"node" : { "id" : "...", "type" : "..." },
+		"dlDst" : "...",
+		"name" : "...",
+		"nwSrc" : "...",
+		"vlanPriority" : "...",
+		"protocol" : "...",
+		"priority" : "...",
+		"vlanId" : "...",
+		"tpDst" : "...",
+		"etherType" : "...",
+		"tpSrc" : "...",
+		"ingressPort" : "...",
+		"idleTimeout" : "...",
+		"dlSrc" : "..." }
+	</programlisting>
+	<para>The following python client writen by Dwcarder describe more specific about
using REST API:<ulink url="https://github.com/dwcarder/python-OpenDaylight/blob/master/OpenDaylight.py">https://github.com/dwcarder/python-OpenDaylight/blob/master/OpenDaylight.py</ulink></para>
+	<para>In project, I learnt how to make HTTP request from CloudStack to ODL for controlling
and managing network services. However, there is a problem that ODL currently don't support
L2 configuration, while integration ODL to CloudStack requires this. I found an incubating
project, led by Brent Salisbury and Evan Zeller from the University of Kentucky, is currently
trying to integrate OpenvSwitch database management protocol to ODL, which will allow ODL
to view, modify and delete OpenvSwitch object such as bridges and ports by way of the OpenvSwitch
databse. In short, this project mainly creates a module acts like OVSDB-client and uses JSON-RPC
for remote management. I talked to them and jumped into this project. Thus, I'll do an extra
work on ODL community to improve ODL Controller support L2 configuration while still integrate
ODL to CloudStack by making a new ODL plugin with the same behavior of NiciraNvp and Ovs.</para>
+	<para>Full information about the incubating project can be found here:<ulink url="https://wiki.opendaylight.org/view/Project_Proposals:OVSDB-Integration">https://wiki.opendaylight.org/view/Project_Proposals:OVSDB-Integration</ulink></para>
+	<para>The next session I will take a short description about XenAPI (also called Xapi),
which applications use to interact virtualization resources in Xen hosts.</para>
+    </section>
+    <section id="xapi">
+	<title>Xen API</title>
+	<para>There are many tool stacks we can use to manage Xen hosts, such as: XL, Xapi,
libvirt or Xend. Of which, Xapi is the default. Xapi (or Xen API) is called from applications
to control and manage virtualization resources in Xen hosts via XML-RPC. Xapi is the core
component of XCP and XenServer and writen by Ocaml language.</para>
+	<para>It's possible to talk directly to Xapi using XML-RPC. This is a way to make
remote procedure calls using http requests. In fact, it's possible to send and receive messages
using telnet but this is not recommended. The XML-RPC calls are the fixed standard, but we
also have bindings to that XML-RPC for Python, C and Java.</para>
+	<para>For example about using XML-RPC calls, I make a simple request written by python
to list all VMs on a Xen host.</para>
+	<para>First thing we need to import XenAPI lib:</para>
+	<programlisting language="Python">
+		>>> import XenAPI
+	</programlisting>
+	<para>Then we have to authenticate to XenServer or XCP addressed from <emphasis>url</emphasis>
with <emphasis>user</emphasis> and <emphasis>password</emphasis></para>
+	<programlisting language="Python">
+		>>> session = XenAPI.Session('https://url')
+		>>> session.login_with_password('user','password')
+	</programlisting>
+	<para>If this works, we've done the hard bit and established communications with our
server. Function bellow will list all Vms on this server.</para>
+	<programlisting language="Python">
+		>>> session.xenapi.VM.get_all()
+	</programlisting>
+	<para>The answer should be something like:</para>
+	<programlisting>
+		['OpaqueRef:7b737e4f-58d8-b493-ea31-31324a2de528', 'OpaqueRef:7237b8af-b80c-c021-fbdc-68146d98d7f5',
.........,	'OpaqueRef:c3b752b9-1926-9ceb-f36a-408497c3478b']
+	</programlisting>
+	<para>Which is a list of strings, each of which represents a unique identifier for
a particular 'object' on the server. In this case of each 'OpaqueRef' represents a virtual
machine. For each VM we can get the name (name_label)</para>
+	<programlisting language="Python">
+		>>> [session.xenapi.VM.get_name_label(x) for x in session.xenapi.VM.get_all()]
+	</programlisting>
+	<para>There are a lot of machines in this list. Some of them however are 'template
Vms', frozen copies which can't actually run, but which can be cloned in oder to make real
virtual machines. We can find out which Vms are templates by calling the VM.get_is_a_template()
function. So let's combinate the two in order to produce a list of all the real Vms on my
server:</para>
+	<programlisting language="Python">
+		>>> [session.xenapi.VM.get_name_label(x) for x in session.xenapi.VM.get_all()
if not session.xenapi.VM.get_is_a_template(x)]
+	</programlisting>
+	<para>The answer should be something like:</para>
+	<programlisting>
+		['Debian Etch 4.0 (2)', 'Debian Etch 4.0 (1)', 'test9', 'test4', 'Control domain on host:
ebony', 'Control domain on host: localhost.localdomain', 'test3', 'Debian Sarge 3.1 (1)',
'test2', 'Debian Etch 4.0 (3)', 'test1', 'test3', 'test7', 'test5']
+	</programlisting>
+	<para>Finally it's only polite to log out of the server. This allows it to garbage
collect the no-longer active session.</para>
+	<programlisting language="Python">
+		>>> session.logout()
+	</programlisting>
+	<para>Full python script can be found here: <ulink url="https://gist.github.com/ngtuna/6094938">Xapi
python client</ulink></para>
+	<para>We can find Xapi source code from: <ulink url="https://github.com/xen-org/xen-api">https://github.com/xen-org/xen-api</ulink></para>
+	<para>Xapi come with some main classes, each of them refer to a virtual resource object
in Xen such as:</para>
+	<itemizedlist>
+		<listitem><para>VM: refer to virtual machine.</para></listitem>
+		<listitem><para>VIF: refer to virtual NIC.</para></listitem>
+		<listitem><para>VDI: refer to virtual volume or hard disk.</para></listitem>
+		<listitem><para>...</para></listitem>
+	</itemizedlist>
+	<para>Full information about Xapi source code we can find here. <ulink url="http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/">http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/</ulink>
Click on each item we can see more detail.</para>
+	<para><emphasis role="bold">Xapi plugin</emphasis></para>
+	<para>Xapi has an extension mechanism that allows one to install a Python script (usually
but it can be any executable) on the Xen host, and then call that through the Xapi. Writing
a Xapi plugin in Python is simplified by using the XenAPIPlugin module, which is by default
installed in dom0 in XCP. In my GsoC project, I have to call some plugin scripts to control
and manage virtual switches. For example, I inserted a new function to get network name-label
in vmops script.</para>
+	<para>Then, we can call it directly from XE command line or via XML-RPC. Here is a
simple call from XE:</para>
+	<programlisting>
+		$xe host-call-plugin host-uuid=<emphasis>host-uuid</emphasis> plugin=vmops
fn=getLabel
+	</programlisting>
+	<para>If the plugins has some arguments, it should be inserted with "args:" keyword.</para>
+	<para>In ACS, almost plugins are called from CitrixResourceBase.java. With my above
function, I inserted a new method into CitrixResourceBase.java and called to the plugin as
below:</para>
+	<programlisting language="Java">
+		private String getLabel() {
+		    	Connection conn = getConnection();
+		    	String result = callHostPlugin(conn, "ovstunnel", "getLabel");
+		    	return result;
+		    }	
+	</programlisting>
+	<para>Of which, Connection class will init a session to Xen host and callHostPlugin
method executes a XML-RPC call to plugin.</para>
+	<para>Note that every Xapi plugin scripts must be placed into /etc/xapi.d/plugins.</para>
+    </section>
+    <section id="whativedone">
+	<title>What I've done</title>
+	<para>In one-and-a-half month, I have understood all of above knowledge and finished
two things:</para>
+	<itemizedlist>
+		<listitem><para>improve gre controller to support XCP.</para></listitem>
+		<listitem><para>re-factor GRE source code by following NiciraNVP plugin design.</para></listitem>
+	</itemizedlist>
+	<para><emphasis role="bold">improve gre controller to support XCP</emphasis></para>
+	<para>From the understanding of how the native SDN works, a small patch has been made
to help it works with Xen Cloud Platform (XCP) version 1.6. Without the patch, this controller
can serve XenServer only, the commercial version of XCP. I did try SDN with XCP and debug
to find out what errors are and why they occur. After some efforts, I figured out following
problems:</para>
+	<itemizedlist>
+		<listitem><para>The SDN controller has to know what interface it'll deploy
GRE tunnels. To do this check, it looks into network to find out the PIF's interface. It has
a network name-label, which user defined in the deploy zone phase. If not, it will be replaced
by a default label. However, XCP's network has no user-defined or default name-label. Therefore
in this step I have made a trick. I used whatever name-label found in the XCP host to bypass
this check.</para></listitem>
+		<listitem><para>When creating an OVS bridge, the controller creates a new dom0
vif, plugs to the bridge and immediately unplugs it. This action aims to ask XenServer create
the bridge without running ovs-vsctl or brctl script. I saw that it is not very important
to XCP hosts and also generates an error from xenopsd daemon, so I ignored this step.</para></listitem>
+		<listitem><para>The script playing a direct role to interact with openvswitch
is ovstunnel. It requires a lib named cloudstack_pluginlib, which does not exist in XCP. Thus,
I inserted this file into copying process from CloudStack to XCP when add-host phase occurs.</para></listitem>
+		<listitem><para>The "setup_ovs_bridge" function in ovstunnel takes a look into
XenServer version to act a blocking IPv6. However, product_version parameter does not exist
on XCP. It uses platform_version parameter instead. So, I decided to ignore this step.</para></listitem>
+	</itemizedlist>
+	<para>The patch is already committed to sdnextensions branch. It is also the primary
branch I have been working on this GSoC period.</para>
+	<para><emphasis role="bold">re-factor GRE source code by following NiciraNVP
plugin design</emphasis></para>
+	<para>GRE source code was re-factored with following changes:</para>
+	<itemizedlist>
+		<listitem><para>add Connectivity service checking: All of L2 configuration
methods now have to check whether Ovs plugin can handle Connectivity service..</para></listitem>
+		<listitem><para>move commands / answers to a new package: com.cloud.agent.api.</para></listitem>
+		<listitem><para>add new NetworkProvider: Ovs.</para></listitem>
+		<listitem><para>add L3 services to Ovs Capabilities: Ovs Capability now is
set enabled to such L3 services as SourceNat, StaticNat, PortForwarding, RedundantRouter,
Gateway. L2 service Connectivity is also set enabled.</para></listitem>
+		<listitem><para>add L3 services prototype code to OvsElement.java</para></listitem>
+	</itemizedlist>
+	<para>With the knowledge about CloudStack's network architecture I have learned and
represented above, I made a patch which permits guest networks can reach each other via private
IPaddress without using VPC mode. Proposal can be found here: <ulink url="https://cwiki.apache.org/confluence/display/CLOUDSTACK/Routing+between+Guest+networks">Routing
between guest networks</ulink></para>
+	<para>In next days, I will done the following things:</para>
+	<itemizedlist>
+		<listitem><para>implement L3 services with Virtual Router.</para></listitem>
+		<listitem><para>improve Ovs to support KVM hypervisor.</para></listitem>
+		<listitem><para>add new ODL plugin using ODL controller to control and manager
network services.</para></listitem>
+	</itemizedlist>
+    </section>
 </section>

http://git-wip-us.apache.org/repos/asf/cloudstack/blob/fd32c33d/docs/en-US/images/l3_services.png
----------------------------------------------------------------------
diff --git a/docs/en-US/images/l3_services.png b/docs/en-US/images/l3_services.png
new file mode 100644
index 0000000..f68aaf3
Binary files /dev/null and b/docs/en-US/images/l3_services.png differ

http://git-wip-us.apache.org/repos/asf/cloudstack/blob/fd32c33d/docs/en-US/images/network_service.png
----------------------------------------------------------------------
diff --git a/docs/en-US/images/network_service.png b/docs/en-US/images/network_service.png
new file mode 100644
index 0000000..95281aa
Binary files /dev/null and b/docs/en-US/images/network_service.png differ

http://git-wip-us.apache.org/repos/asf/cloudstack/blob/fd32c33d/docs/en-US/images/odl_structure.jpg
----------------------------------------------------------------------
diff --git a/docs/en-US/images/odl_structure.jpg b/docs/en-US/images/odl_structure.jpg
new file mode 100644
index 0000000..08e0012
Binary files /dev/null and b/docs/en-US/images/odl_structure.jpg differ


Mime
View raw message