cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Murali Reddy (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CLOUDSTACK-6715) [SDN] Inconsistency in ovs-flow table after vm migration from one host to another
Date Tue, 17 Jun 2014 09:22:02 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Murali Reddy resolved CLOUDSTACK-6715.
--------------------------------------

    Resolution: Duplicate


Root cause is same as 6779. There is crash of OVSswitch daemon which results in bridges getting
recreated. Once bridge is recreated we loose all the flow rules configured for the brdige.

> [SDN] Inconsistency in ovs-flow table after vm migration from one host to another 
> ----------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-6715
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6715
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server, Network Controller
>    Affects Versions: 4.4.0
>         Environment: Latest build from 4.4 with commit e6961fd21bb6d793302c234d0f409f66dc498072
>            Reporter: Sanjeev N
>            Assignee: Murali Reddy
>            Priority: Critical
>             Fix For: 4.4.0
>
>         Attachments: management-server.rar, ovstunnel-host13.log, ovstunnel-host14.log
>
>
> [SDN] Inconsistency in ovs-flow table after vm migration from one host to another 
> Steps to reproduce:
> ================
> 1.Bring up CS in advanced zone with two xen hosts in a cluster
> 2.Create physical network with GRE isolation
> 3.Create isolated network with OVS provider 
> 4.Deploy few vms in the network and make sure that all the vms(including VR) are deployed
on only one host.
> 5.Now migrate one vm to another host in the cluster and verify flow tables for this isolated
network bridge on both the xen hosts
> Result:
> =======
> Inconsistency in flow tables on both the xen hosts
> Following it the flow table from the host before vm migration (All the vms and VR were
only on this host before vm migration):
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=1173.14s, table=0, n_packets=0, n_bytes=0, priority=1100,dl_dst=ff:ff:ff:ff:ff:ff
actions=output:1,output:2
>  cookie=0x0, duration=1173.15s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24
actions=NORMAL
>  cookie=0x0, duration=1203.276s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=1,nw_dst=224.0.0.0/24
actions=NORMAL
>  cookie=0x0, duration=1226.258s, table=0, n_packets=901, n_bytes=85612, priority=0 actions=NORMAL
>  cookie=0x0, duration=1173.129s, table=0, n_packets=0, n_bytes=0, priority=1100,ip,nw_dst=224.0.0.0/24
actions=output:1,output:2
>  cookie=0x0, duration=1203.286s, table=0, n_packets=0, n_bytes=0, priority=1200,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff
actions=NORMAL
>  cookie=0x0, duration=1173.167s, table=0, n_packets=7, n_bytes=1494, priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff
actions=NORMAL
> Flow table after vm migration on the same host:
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=992.789s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
> [root@xen-host-14 ~]#
> Ports info on the bridge after vm migration:
> [root@xen-host-14 ~]# ovs-vsctl list-ports xapi4
> t986-2-1
> vif11.0
> vif12.0
> Flow table on the target host where vm was migrated to:
> [root@xen-host-13 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=1025.129s, table=0, n_packets=0, n_bytes=0, priority=1100,dl_dst=ff:ff:ff:ff:ff:ff
actions=output:2
>  cookie=0x0, duration=1025.139s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24
actions=NORMAL
>  cookie=0x0, duration=1032.932s, table=0, n_packets=0, n_bytes=0, priority=1000,ip,in_port=1,nw_dst=224.0.0.0/24
actions=drop
>  cookie=0x0, duration=1033.247s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
>  cookie=0x0, duration=1025.119s, table=0, n_packets=0, n_bytes=0, priority=1100,ip,nw_dst=224.0.0.0/24
actions=output:2
>  cookie=0x0, duration=1032.942s, table=0, n_packets=0, n_bytes=0, priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff
actions=drop
>  cookie=0x0, duration=1025.148s, table=0, n_packets=1, n_bytes=42, priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff
actions=NORMAL
> [root@xen-host-13 ~]# ovs-vsctl list-ports xapi4
> t986-1-2
> vif17.0
> Following is the log snippet from both the hosts during vm migration (tunnel creation
during vm migration):
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS enter  create_tunnel ####
> 2014-05-20 11:31:21    DEBUG [root] Creating tunnel from host 2 to host 1 with GRE key
986
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=30',
'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge', 'xapi4', 'name']
> 2014-05-20 11:31:21    DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi4',
't986-2-1', '--', 'set', 'interface', 't986-2-1', 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.13']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'port', 't986-2-1',
'interfaces']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:key']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22    DEBUG [root] Tunnel interface validated:['/usr/bin/ovs-vsctl',
'get', 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'01be01e9-c4b1-4b90-9ac0-199f2c797719', 'ofport']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:22    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:22    DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:22    DEBUG [root] Successfully created tunnel from host 2 to host 1
with GRE key 986
> 2014-05-20 11:31:22    DEBUG [root] #### VMOPS exit  create_tunnel ####
> 2014-05-20 11:31:29    DEBUG [root] Executing:['cat', '/etc/xensource/network.conf']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'iface-to-br', 'vif19.0']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'list-ports', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
't986-2-1', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif11.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif12.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif19.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'del-flows', 'xapi4',
',in_port=3']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:1,output:2']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:1,output:2']
> ovstunnel log from the host where vm has been migrated to:
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS enter  setup_ovs_bridge ####
> 2014-05-20 11:31:21    DEBUG [root] About to manually create the bridge:xapi4
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--', '--may-exist',
'add-br', 'xapi4', '--', 'set', 'bridge', 'xapi4', 'other_config:gre_key=OVSTunnel986']
> 2014-05-20 11:31:21    DEBUG [root] Bridge has been manually created:
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set', 'Bridge',
'xapi4', 'external_ids:xs-network-uuid=98636962-280c-bb8f-1e57-d79bcc295c1f']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set', 'Bridge',
'xapi4', 'stp_enable=true']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'bridge',
'xapi4', 'other_config:gre_key']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-set',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'other-config:is-ovs-tun-network=True']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=ovs-host-setup']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-set',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'other-config:ovs-host-setup=1,2']
> 2014-05-20 11:31:21    DEBUG [root] Setup_ovs_bridge completed with result:SUCCESS:xapi4
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS exit  setup_ovs_bridge ####
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS enter  create_tunnel ####
> 2014-05-20 11:31:21    DEBUG [root] Creating tunnel from host 1 to host 2 with GRE key
986
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=30',
'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge', 'xapi4', 'name']
> 2014-05-20 11:31:21    DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi4',
't986-1-2', '--', 'set', 'interface', 't986-1-2', 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.14']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'port', 't986-1-2',
'interfaces']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:key']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21    DEBUG [root] Tunnel interface validated:['/usr/bin/ovs-vsctl',
'get', 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'ofport']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:21    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:21    DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:21    DEBUG [root] Successfully created tunnel from host 1 to host 2
with GRE key 986
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS exit  create_tunnel ####
> 2014-05-20 11:31:29    DEBUG [root] Executing:['cat', '/etc/xensource/network.conf']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'iface-to-br', 'vif17.0']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'list-ports', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
't986-1-2', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif17.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff,actions=NORMAL']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,ip,nw_dst=224.0.0.0/24,actions=NORMAL']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:2']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:2']
> ~



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message