cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Murali Reddy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-6755) [OVS] Can't create more than 7 GRE tunnel networks in xen cluster
Date Thu, 28 Aug 2014 07:01:05 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113448#comment-14113448
] 

Murali Reddy commented on CLOUDSTACK-6755:
------------------------------------------

Problem;
-----------
 GRE tunnel networks are internal networks in XenServer. A XenServer network created on a
XenServer, is visible across the hosts in the clusters. Only on the first host on which a
VM is launched there is a bridge created automatically. When a VM is launched on a different
host in the cluster, but on the same network, VM start will fail with VM_REQUIRES_NETWORK
exception as bridge is not automatically created by the Xenserver. To overcome this issue,
there is a hack put into CloudStack, so that bridge is create for internal network that is
backing GRE tunnel network. A vif is created on to the network and plugged in to dom0 which
will result in the bridge getting created. But this approach will results in only creating
7 tunnel networks as for any dom0/domu only 7 vif's can be connected

Root Cause Analysis:
--------------------------
Root cause is using the dom0 vif connected to a GRE tunnel network as a forcing function to
create bridge on the host. this VID is never deleted resulting in accumulation of vif's on
dom0.

Proposed Solution:
------------------------

Proposed solution is to ensure VIF gets deleted after bridge is created and at least one VIF
(from the domU/guest VM's) is connected to the bridge. 

QA steps:
-------------

To verify the fix works. 

  - Please create more that 7 GRE tunnel networks and verify CloudStack no longer restricts
only 7 GRE tunnel networks.
  - also verify there are no unnecessary VIF's (that are connected to GRE tunnel networks)
attached to dom0.





> [OVS] Can't create more than 7 GRE tunnel networks in xen cluster
> -----------------------------------------------------------------
>
>                 Key: CLOUDSTACK-6755
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6755
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server, Network Controller, XenServer
>    Affects Versions: 4.4.0
>         Environment: Latest build from 4.4 with commit e6961fd21bb6d793302c234d0f409f66dc498072
>            Reporter: Sanjeev N
>            Assignee: Murali Reddy
>            Priority: Blocker
>              Labels: ovs
>             Fix For: 4.4.0
>
>         Attachments: management-server.rar, ovstunnel.log, xensource.log
>
>
> [OVS] Can't create more than 7 GRE tunnel networks in xen cluster
> Steps to reproduce:
> ==================
> 1.Bring up CS in advanced zone with Xen cluster
> 2.Create physical network with GRE isolation
> 3.Create isolated network offering with virtual networking and OVS as the service provider
> 4.Create 8 guest networks with above offering and deploy vms in each network
> Result:
> ======
> CS would create 7 networks successfully and deploying vm in 8th network would fail because
Tunnel network creation would fail for 8th network
> Following is the log snippet from MS log:
> 2014-05-23 10:18:12,784 DEBUG [c.c.n.e.OvsElement] (Work-Job-Executor-89:job-253/job-254
ctx-800616c3) Checking if OvsElement can handle service Connectivity on network test1-ovs
> 2014-05-23 10:18:12,789 DEBUG [c.c.n.o.OvsTunnelManagerImpl] (Work-Job-Executor-89:job-253/job-254
ctx-800616c3) Creating tunnels with OVS tunnel manager
> 2014-05-23 10:18:12,800 DEBUG [c.c.n.o.OvsTunnelManagerImpl] (Work-Job-Executor-89:job-253/job-254
ctx-800616c3) Ask host 1 to configure bridge for network:227
> 2014-05-23 10:18:12,807 DEBUG [c.c.a.t.Request] (Work-Job-Executor-89:job-253/job-254
ctx-800616c3) Seq 1-6553863357730929719: Sending  { Cmd , MgmtId: 7332683579487, via: 1(xen-host-13),
Ver: v1, Flags: 100111, [{"com.cloud.agent.api.OvsSetupBridgeCommand":{"name":"OVSTunnel991","hostId":1,"networkId":227,"wait":0}}]
}
> 2014-05-23 10:18:12,808 DEBUG [c.c.a.t.Request] (Work-Job-Executor-89:job-253/job-254
ctx-800616c3) Seq 1-6553863357730929719: Executing:  { Cmd , MgmtId: 7332683579487, via: 1(xen-host-13),
Ver: v1, Flags: 100111, [{"com.cloud.agent.api.OvsSetupBridgeCommand":{"name":"OVSTunnel991","hostId":1,"networkId":227,"wait":0}}]
}
> 2014-05-23 10:18:12,808 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-268:ctx-46e3abbc)
Seq 1-6553863357730929719: Executing request
> 2014-05-23 10:18:12,991 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-268:ctx-46e3abbc)
Create a vif on dom0 for tunnel network for account OVSTunnel991
> 2014-05-23 10:18:13,056 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-268:ctx-46e3abbc)
createTunnelNetwork failed
> com.cloud.utils.exception.CloudRuntimeException: Could not find available VIF slot in
VM with name: Control domain on host: xen-host-13
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getLowestAvailableVIFDeviceNum(CitrixResourceBase.java:3990)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.enableXenServerNetwork(CitrixResourceBase.java:928)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.findOrCreateTunnelNetwork(CitrixResourceBase.java:1003)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5268)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:528)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
>         at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-05-23 10:18:13,175 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-268:ctx-46e3abbc)
Create a vif on dom0 for tunnel network for account OVSTunnel991
> 2014-05-23 10:18:13,239 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-268:ctx-46e3abbc)
createTunnelNetwork failed
> com.cloud.utils.exception.CloudRuntimeException: Could not find available VIF slot in
VM with name: Control domain on host: xen-host-13
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.getLowestAvailableVIFDeviceNum(CitrixResourceBase.java:3990)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.enableXenServerNetwork(CitrixResourceBase.java:928)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.findOrCreateTunnelNetwork(CitrixResourceBase.java:1007)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configureTunnelNetwork(CitrixResourceBase.java:1022)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5269)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:528)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
>         at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-05-23 10:18:13,242 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-268:ctx-46e3abbc)
createandConfigureTunnelNetwork failed
> java.lang.NullPointerException
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configureTunnelNetwork(CitrixResourceBase.java:1025)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5269)
>         at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:528)
>         at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
>         at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
>         at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> 2014-05-23 10:18:13,246 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-268:ctx-46e3abbc)
OVS Bridge configured
> Observations:
> ===========
> On xenserver dom0 vm we are creating vif for every GRE tunnel network. But dom0 has a
limitation of 7 vifs. So creating 8th tunnel network is failing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message