cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjeev N (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-6715) [SDN] Inconsistency in ovs-flow table after vm migration from one host to another
Date Tue, 20 May 2014 09:40:38 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sanjeev N updated CLOUDSTACK-6715:
----------------------------------

    Assignee: Murali Reddy

> [SDN] Inconsistency in ovs-flow table after vm migration from one host to another 
> ----------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-6715
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6715
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server, Network Controller
>    Affects Versions: 4.4.0
>         Environment: Latest build from 4.4 with commit e6961fd21bb6d793302c234d0f409f66dc498072
>            Reporter: Sanjeev N
>            Assignee: Murali Reddy
>            Priority: Critical
>             Fix For: 4.4.0
>
>         Attachments: management-server.rar, ovstunnel-host13.log, ovstunnel-host14.log
>
>
> [SDN] Inconsistency in ovs-flow table after vm migration from one host to another 
> Steps to reproduce:
> ================
> 1.Bring up CS in advanced zone with two xen hosts in a cluster
> 2.Create physical network with GRE isolation
> 3.Create isolated network with OVS provider 
> 4.Deploy few vms in the network and make sure that all the vms(including VR) are deployed
on only one host.
> 5.Now migrate one vm to another host in the cluster and verify flow tables for this isolated
network bridge on both the xen hosts
> Result:
> =======
> Inconsistency in flow tables on both the xen hosts
> Following it the flow table from the host before vm migration (All the vms and VR were
only on this host before vm migration):
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=1173.14s, table=0, n_packets=0, n_bytes=0, priority=1100,dl_dst=ff:ff:ff:ff:ff:ff
actions=output:1,output:2
>  cookie=0x0, duration=1173.15s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24
actions=NORMAL
>  cookie=0x0, duration=1203.276s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=1,nw_dst=224.0.0.0/24
actions=NORMAL
>  cookie=0x0, duration=1226.258s, table=0, n_packets=901, n_bytes=85612, priority=0 actions=NORMAL
>  cookie=0x0, duration=1173.129s, table=0, n_packets=0, n_bytes=0, priority=1100,ip,nw_dst=224.0.0.0/24
actions=output:1,output:2
>  cookie=0x0, duration=1203.286s, table=0, n_packets=0, n_bytes=0, priority=1200,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff
actions=NORMAL
>  cookie=0x0, duration=1173.167s, table=0, n_packets=7, n_bytes=1494, priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff
actions=NORMAL
> Flow table after vm migration on the same host:
> [root@xen-host-14 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=992.789s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
> [root@xen-host-14 ~]#
> Ports info on the bridge after vm migration:
> [root@xen-host-14 ~]# ovs-vsctl list-ports xapi4
> t986-2-1
> vif11.0
> vif12.0
> Flow table on the target host where vm was migrated to:
> [root@xen-host-13 ~]# ovs-ofctl dump-flows xapi4
> NXST_FLOW reply (xid=0x4):
>  cookie=0x0, duration=1025.129s, table=0, n_packets=0, n_bytes=0, priority=1100,dl_dst=ff:ff:ff:ff:ff:ff
actions=output:2
>  cookie=0x0, duration=1025.139s, table=0, n_packets=0, n_bytes=0, priority=1200,ip,in_port=2,nw_dst=224.0.0.0/24
actions=NORMAL
>  cookie=0x0, duration=1032.932s, table=0, n_packets=0, n_bytes=0, priority=1000,ip,in_port=1,nw_dst=224.0.0.0/24
actions=drop
>  cookie=0x0, duration=1033.247s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
>  cookie=0x0, duration=1025.119s, table=0, n_packets=0, n_bytes=0, priority=1100,ip,nw_dst=224.0.0.0/24
actions=output:2
>  cookie=0x0, duration=1032.942s, table=0, n_packets=0, n_bytes=0, priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff
actions=drop
>  cookie=0x0, duration=1025.148s, table=0, n_packets=1, n_bytes=42, priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff
actions=NORMAL
> [root@xen-host-13 ~]# ovs-vsctl list-ports xapi4
> t986-1-2
> vif17.0
> Following is the log snippet from both the hosts during vm migration (tunnel creation
during vm migration):
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS enter  create_tunnel ####
> 2014-05-20 11:31:21    DEBUG [root] Creating tunnel from host 2 to host 1 with GRE key
986
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=30',
'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge', 'xapi4', 'name']
> 2014-05-20 11:31:21    DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi4',
't986-2-1', '--', 'set', 'interface', 't986-2-1', 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.13']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'port', 't986-2-1',
'interfaces']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:key']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22    DEBUG [root] Tunnel interface validated:['/usr/bin/ovs-vsctl',
'get', 'interface', '01be01e9-c4b1-4b90-9ac0-199f2c797719', 'options:remote_ip']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'01be01e9-c4b1-4b90-9ac0-199f2c797719', 'ofport']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:22    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:22    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=4,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:22    DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:22    DEBUG [root] Successfully created tunnel from host 2 to host 1
with GRE key 986
> 2014-05-20 11:31:22    DEBUG [root] #### VMOPS exit  create_tunnel ####
> 2014-05-20 11:31:29    DEBUG [root] Executing:['cat', '/etc/xensource/network.conf']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'iface-to-br', 'vif19.0']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'list-ports', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
't986-2-1', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif11.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif12.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif19.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'del-flows', 'xapi4',
',in_port=3']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:1,output:2']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:1,output:2']
> ovstunnel log from the host where vm has been migrated to:
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS enter  setup_ovs_bridge ####
> 2014-05-20 11:31:21    DEBUG [root] About to manually create the bridge:xapi4
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--', '--may-exist',
'add-br', 'xapi4', '--', 'set', 'bridge', 'xapi4', 'other_config:gre_key=OVSTunnel986']
> 2014-05-20 11:31:21    DEBUG [root] Bridge has been manually created:
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set', 'Bridge',
'xapi4', 'external_ids:xs-network-uuid=98636962-280c-bb8f-1e57-d79bcc295c1f']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'set', 'Bridge',
'xapi4', 'stp_enable=true']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'bridge',
'xapi4', 'other_config:gre_key']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-set',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'other-config:is-ovs-tun-network=True']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=ovs-host-setup']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-set',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'other-config:ovs-host-setup=1,2']
> 2014-05-20 11:31:21    DEBUG [root] Setup_ovs_bridge completed with result:SUCCESS:xapi4
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS exit  setup_ovs_bridge ####
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS enter  create_tunnel ####
> 2014-05-20 11:31:21    DEBUG [root] Creating tunnel from host 1 to host 2 with GRE key
986
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', '--timeout=30',
'wait-until', 'bridge', 'xapi4', '--', 'get', 'bridge', 'xapi4', 'name']
> 2014-05-20 11:31:21    DEBUG [root] bridge xapi4 for creating tunnel - VERIFIED
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'add-port', 'xapi4',
't986-1-2', '--', 'set', 'interface', 't986-1-2', 'type=gre', 'options:key=986', 'options:remote_ip=10.147.40.14']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'port', 't986-1-2',
'interfaces']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:key']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21    DEBUG [root] Tunnel interface validated:['/usr/bin/ovs-vsctl',
'get', 'interface', 'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'options:remote_ip']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'interface',
'e5d8a954-a482-42c2-88a6-7294ad62c7f8', 'ofport']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:21    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,dl_dst=ff:ff:ff:ff:ff:ff,actions=drop']
> 2014-05-20 11:31:21    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1000,in_port=1,ip,nw_dst=224.0.0.0/24,actions=drop']
> 2014-05-20 11:31:21    DEBUG [root] Broadcast drop rules added
> 2014-05-20 11:31:21    DEBUG [root] Successfully created tunnel from host 1 to host 2
with GRE key 986
> 2014-05-20 11:31:21    DEBUG [root] #### VMOPS exit  create_tunnel ####
> 2014-05-20 11:31:29    DEBUG [root] Executing:['cat', '/etc/xensource/network.conf']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'iface-to-br', 'vif17.0']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-list',
'bridge=xapi4', '--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-tun-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/opt/xensource/bin/xe', 'network-param-get',
'uuid=98636962-280c-bb8f-1e57-d79bcc295c1f', 'param-name=other-config', 'param-key=is-ovs-vpc-distributed-vr-network',
'--minimal']
> 2014-05-20 11:31:29    DEBUG [root] The command exited with the error code: 1 (stderr
output:Error: Key is-ovs-vpc-distributed-vr-network not found in map
> )
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'br-to-vlan', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'list-ports', 'xapi4']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
't986-1-2', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-vsctl', 'get', 'Interface',
'vif17.0', 'ofport']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,dl_dst=ff:ff:ff:ff:ff:ff,actions=NORMAL']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1200,in_port=2,ip,nw_dst=224.0.0.0/24,actions=NORMAL']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,dl_dst=ff:ff:ff:ff:ff:ff,actions=output:2']
> 2014-05-20 11:31:29    DEBUG [root] Executing:['/usr/bin/ovs-ofctl', 'add-flow', 'xapi4',
'hard_timeout=0,idle_timeout=0,priority=1100,ip,nw_dst=224.0.0.0/24,actions=output:2']
> ~



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message