cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "venkata swamybabu budumuru (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (CLOUDSTACK-4080) Router VM is stopped by scavenger thread as part of DeployVMCmd if the network.gc is set to low value like "10" seconds
Date Wed, 07 Aug 2013 07:37:50 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

venkata swamybabu budumuru closed CLOUDSTACK-4080.
--------------------------------------------------


Verified this issue on the latest build and I don't see this issue anymore. closing the bug.
                
> Router VM is stopped by scavenger thread as part of DeployVMCmd if the network.gc is
set to low value like "10" seconds
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4080
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4080
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Network Controller
>    Affects Versions: 4.2.0
>         Environment: commit # 7522f811672f66bc0cc13a33f4f3737ef03f22af
>            Reporter: venkata swamybabu budumuru
>            Assignee: Alena Prokharchyk
>            Priority: Critical
>             Fix For: 4.2.0
>
>         Attachments: logs.tgz
>
>
> Steps to reproduce:
> 1. Have latest CloudStack setup with at least one advanced zone.
> 2. Make sure network.gc.interval and wait are set to "10" seconds
> 3. Have at least one network offering of type "isolated" and with all services enabled
where LB is provided by NS and other services are provided by VR.
> mysql> select * from network_offerings where id=15\G
> *************************** 1. row ***************************
>                        id: 15
>                      name: NetworkOffering with NS
>                      uuid: 4aaf5c58-6d45-4213-8c26-0b2b6f6792c5
>               unique_name: NetworkOffering with NS
>              display_text: NetworkOffering with NS
>                   nw_rate: NULL
>                   mc_rate: 10
>              traffic_type: Guest
>                      tags: NULL
>               system_only: 0
>              specify_vlan: 0
>       service_offering_id: NULL
>             conserve_mode: 0
>                   created: 2013-08-05 07:30:38
>                   removed: NULL
>                   default: 0
>              availability: Optional
>      dedicated_lb_service: 0
> shared_source_nat_service: 0
>                  sort_key: 0
>  redundant_router_service: 0
>                     state: Enabled
>                guest_type: Isolated
>        elastic_ip_service: 0
>   eip_associate_public_ip: 0
>        elastic_lb_service: 0
>         specify_ip_ranges: 0
>                    inline: 0
>             is_persistent: 0
>               internal_lb: 0
>                 public_lb: 1
>     egress_default_policy: 1
>    concurrent_connections: NULL
> mysql> select * from ntwk_offering_service_map where network_offering_id=15;
> +----+---------------------+----------------+---------------+---------------------+
> | id | network_offering_id | service        | provider      | created             |
> +----+---------------------+----------------+---------------+---------------------+
> | 58 |                  15 | Dhcp           | VirtualRouter | 2013-08-05 07:30:38 |
> | 55 |                  15 | Dns            | VirtualRouter | 2013-08-05 07:30:38 |
> | 60 |                  15 | Firewall       | VirtualRouter | 2013-08-05 07:30:38 |
> | 59 |                  15 | Lb             | Netscaler     | 2013-08-05 07:30:38 |
> | 54 |                  15 | PortForwarding | VirtualRouter | 2013-08-05 07:30:38 |
> | 56 |                  15 | SourceNat      | VirtualRouter | 2013-08-05 07:30:38 |
> | 53 |                  15 | StaticNat      | VirtualRouter | 2013-08-05 07:30:38 |
> | 57 |                  15 | UserData       | VirtualRouter | 2013-08-05 07:30:38 |
> | 61 |                  15 | Vpn            | VirtualRouter | 2013-08-05 07:30:38 |
> +----+---------------------+----------------+---------------+---------------------+
> mysql> select * from host_details where host_id=4;
> +----+---------+-------------------+-------------------------------------------+
> | id | host_id | name              | value                                     |
> +----+---------+-------------------+-------------------------------------------+
> | 13 |       4 | deviceName        | NetscalerVPXLoadBalancer                  |
> | 11 |       4 | guid              | 1cf71bde-3994-42eb-80e0-046278a1763d      |
> | 21 |       4 | ip                | 10.147.60.26                              |
> | 19 |       4 | lbdevicededicated | false                                     |
> | 23 |       4 | lbdeviceid        | 1                                         |
> | 16 |       4 | name              | 201-NetscalerVPXLoadBalancer-10.147.60.26 |
> | 17 |       4 | numretries        | 2                                         |
> | 18 |       4 | password          | ck3EWqTylg79ZMj4gG2sHA==                  |
> | 20 |       4 | physicalNetworkId | 201                                       |
> | 15 |       4 | privateinterface  | 1/2                                       |
> | 14 |       4 | publicinterface   | 1/3                                       |
> | 12 |       4 | username          | nsroot                                    |
> | 22 |       4 | zoneId            | 2                                         |
> +----+---------+-------------------+-------------------------------------------+
> 4. As a non-ROOT domain user, Try to deploy a VM using the network that is created using
the above n/w offering.
> mysql> select * from networks where id=242\G
> *************************** 1. row ***************************
>                    id: 242
>                  name: test
>                  uuid: c8028134-77ab-415a-bdb9-d9378754479b
>          display_text: test
>          traffic_type: Guest
> broadcast_domain_type: Vlan
>         broadcast_uri: NULL
>               gateway: 10.0.48.1
>                  cidr: 10.0.48.0/20
>                  mode: Dhcp
>   network_offering_id: 15
>   physical_network_id: 200
>        data_center_id: 1
>             guru_name: ExternalGuestNetworkGuru
>                 state: Allocated
>               related: 242
>             domain_id: 2
>            account_id: 4
>                  dns1: 10.103.128.16
>                  dns2: NULL
>             guru_data: NULL
>            set_fields: 0
>              acl_type: Account
>        network_domain: cs4cloud.internal
>        reservation_id: 803e1334-ed30-4980-a6f2-299427724bb9
>            guest_type: Isolated
>      restart_required: 0
>               created: 2013-08-05 11:27:19
>               removed: NULL
>     specify_ip_ranges: 0
>                vpc_id: NULL
>           ip6_gateway: NULL
>              ip6_cidr: NULL
>          network_cidr: NULL
>       display_network: 1
>        network_acl_id: NULL
> Observations:
> (i) deployVMCmd goes fine without any issues but, network scavenger is going and shutting
down the network immediately after startAnswer for userVM
> (ii) Here is the deployVMCmd :
> 2013-08-05 16:57:19,427 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===END===
 10.252.192.25 -- GET  command=deployVirtualMachine&zoneId=7b6b3c07-7e33-483f-b2a1-2f89a0d9ff96&templateId=4643adee-fd8e-11e2-9c07-069f2c0000aa&hypervisor=KVM&serviceOfferingId=d42e0af6-370b-4a4f-a318-98d1d2a9a8e3&networkIds=c8028134-77ab-415a-bdb9-d9378754479b&displayname=test&name=test&response=json&sessionkey=YGtsbnrjR7V3vmhttXR20I2v8L0%3D&_=1375702049488
> (iii) The above command initiated a router VM deployment
> 2013-08-05 16:57:25,237 DEBUG [agent.transport.Request] (Job-Executor-8:job-59 = [ 17db0422-7b06-4bb6-b78a-ae9728bd26d1
]) Seq 1-1321009484: Sending  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StartCommand":{"vm":{"id":14,"name":"r-14-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-14-VM eth2ip=10.147.44.67 eth2mask=255.255.255.0
gateway=10.147.44.1 eth0ip=10.0.48.1 eth0mask=255.255.240.0 domain=cs4cloud.internal dhcprange=10.0.48.1
eth1ip=169.254.3.57 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.103.128.16","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"821327d55071659e","params":{},"uuid":"3980de9d-1b91-4e7c-ae36-2b9a9d08ef38","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a11a7197-2aed-40e6-9569-6787c577ab2c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5458182e-bfcb-351c-97ed-e7223bca2b8e","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/swamy/primary.campo.kvm.1.zone","port":2049}},"name":"ROOT-14","size":276162048,"path":"458958b5-5497-477e-9e53-1227a0187688","volumeId":17,"vmName":"r-14-VM","accountId":4,"format":"QCOW2","id":17,"hypervisorType":"None"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"65fd9e0c-5b15-4181-9ab1-4a13fea4739e","ip":"10.147.44.67","netmask":"255.255.255.0","gateway":"10.147.44.1","mac":"06:2a:3a:00:00:12","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://44","isolationUri":"vlan://44","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"d2a0ff89-a6ce-4005-a875-d36787318342","ip":"10.0.48.1","netmask":"255.255.240.0","gateway":"10.0.48.1","mac":"02:00:6b:6d:00:02","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://909","isolationUri":"vlan://909","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"0a187643-dc28-48b5-bf99-8acbe5f753c2","ip":"169.254.3.57","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:03:39","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.147.40.11","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.3.57","port":3922,"interval":6,"retries":100,"name":"r-14-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.ip":"169.254.3.57","router.name":"r-14-VM"},"wait":0}},{},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":4,"publicIp":"10.147.44.67","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"44","vlanGateway":"10.147.44.1","vlanNetmask":"255.255.255.0","vifMacAddress":"06:b0:5a:00:00:12","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.0.48.1","zone.network.type":"Advanced","router.ip":"169.254.3.57","router.name":"r-14-VM"},"wait":0}}]
}
> (iv) Router VM started successfully
> 2013-08-05 16:58:40,412 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null)
Seq 1-1321009484: Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":14,"name":"r-14-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-14-VM eth2ip=10.147.44.67 eth2mask=255.255.255.0
gateway=10.147.44.1 eth0ip=10.0.48.1 eth0mask=255.255.240.0 domain=cs4cloud.internal dhcprange=10.0.48.1
eth1ip=169.254.3.57 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=10.103.128.16","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"821327d55071659e","vncAddr":"10.147.40.11","params":{},"uuid":"3980de9d-1b91-4e7c-ae36-2b9a9d08ef38","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"a11a7197-2aed-40e6-9569-6787c577ab2c","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5458182e-bfcb-351c-97ed-e7223bca2b8e","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/swamy/primary.campo.kvm.1.zone","port":2049}},"name":"ROOT-14","size":276162048,"path":"458958b5-5497-477e-9e53-1227a0187688","volumeId":17,"vmName":"r-14-VM","accountId":4,"format":"QCOW2","id":17,"hypervisorType":"None"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"65fd9e0c-5b15-4181-9ab1-4a13fea4739e","ip":"10.147.44.67","netmask":"255.255.255.0","gateway":"10.147.44.1","mac":"06:2a:3a:00:00:12","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://44","isolationUri":"vlan://44","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"d2a0ff89-a6ce-4005-a875-d36787318342","ip":"10.0.48.1","netmask":"255.255.240.0","gateway":"10.0.48.1","mac":"02:00:6b:6d:00:02","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://909","isolationUri":"vlan://909","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"0a187643-dc28-48b5-bf99-8acbe5f753c2","ip":"169.254.3.57","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:03:39","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack
Release 4.2.0 Thu Jun 13 04:15:09 UTC 2013","scriptsVersion":"0026a7d7d957616f59bdeab0c49258bb","result":true,"details":"Cloudstack
Release 4.2.0 Thu Jun 13 04:15:09 UTC 2013&0026a7d7d957616f59bdeab0c49258bb","wait":0}},{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-14-VM","bytesSent":0,"bytesReceived":0,"result":true,"wait":0}},{"com.cloud.agent.api.routing.IpAssocAnswer":{"results":["10.147.44.67
- success"],"result":true,"wait":0}}] }
> (v) After the router VM is up, it triggered startComand for userVM and that as well went
fine.
> 2013-08-05 16:58:43,001 DEBUG [agent.transport.Request] (AgentManager-Handler-14:null)
Seq 1-1321009496: Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":13,"name":"i-4-13-VM","type":"User","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
5.5 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"b1f3f5cddd630be3","vncAddr":"10.147.40.11","params":{},"uuid":"773ccd08-cfef-41ad-8c43-30a169630d0b","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"ffa1c5ad-cdca-40dc-93bd-03c1dbdd4caf","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5458182e-bfcb-351c-97ed-e7223bca2b8e","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/swamy/primary.campo.kvm.1.zone","port":2049}},"name":"ROOT-13","size":8589934592,"path":"ad4b1306-a145-4944-b296-775671c9624b","volumeId":16,"vmName":"i-4-13-VM","accountId":4,"format":"QCOW2","id":16,"hypervisorType":"None"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"302aa348-f8aa-4872-ad59-142500ed9f63","ip":"10.0.49.81","netmask":"255.255.240.0","gateway":"10.0.48.1","mac":"02:00:1a:0c:00:01","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://909","isolationUri":"vlan://909","isSecurityGroupEnabled":false}]},"result":true,"wait":0}}]
}
> (vi) One additional observation here : After the router VM started, I could see the nics.state
as "Allocated" for userVM for sometime.
> *************************** 33. row ***************************
>             id: 33
>           uuid: 302aa348-f8aa-4872-ad59-142500ed9f63
>    instance_id: 13
>    mac_address: 02:00:1a:0c:00:01
>    ip4_address: 10.0.49.81
>        netmask: NULL
>        gateway: NULL
>        ip_type: Ip4
>  broadcast_uri: NULL
>     network_id: 242
>           mode: Dhcp
>          state: Allocated
>       strategy: Start
>  reserver_name: ExternalGuestNetworkGuru
> reservation_id: NULL
>      device_id: 0
>    update_time: 2013-08-05 16:57:19
>  isolation_uri: NULL
>    ip6_address: NULL
>    default_nic: 1
>        vm_type: User
>        created: 2013-08-05 11:27:19
>        removed: NULL
>    ip6_gateway: NULL
>       ip6_cidr: NULL
>   secondary_ip: 0
>    display_nic: 1
> (vii) Once the UserVM is up then I see the nics.state for userVM nic as "Reserved"
> mysql> select * from nics where instance_id=13\G
> *************************** 1. row ***************************
>             id: 33
>           uuid: 302aa348-f8aa-4872-ad59-142500ed9f63
>    instance_id: 13
>    mac_address: 02:00:1a:0c:00:01
>    ip4_address: 10.0.49.81
>        netmask: 255.255.240.0
>        gateway: 10.0.48.1
>        ip_type: Ip4
>  broadcast_uri: vlan://909
>     network_id: 242
>           mode: Dhcp
>          state: Reserved
>       strategy: Start
>  reserver_name: ExternalGuestNetworkGuru
> reservation_id: 803e1334-ed30-4980-a6f2-299427724bb9
>      device_id: 0
>    update_time: 2013-08-05 16:58:40
>  isolation_uri: vlan://909
>    ip6_address: NULL
>    default_nic: 1
>        vm_type: User
>        created: 2013-08-05 11:27:19
>        removed: NULL
>    ip6_gateway: NULL
>       ip6_cidr: NULL
>   secondary_ip: 0
>    display_nic: 1
> 1 row in set (0.00 sec)
> (viii) As soon the startAnswer Comes for the userVM, I see that there is network shutdown
initiated by network scavenger thread.
> Here is the snippet from mgmt server logs.
> 013-08-05 16:58:45,756 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
SeqA 2-790: Processing Seq 2-790:  { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
 \"connections\": []\n}","wait":0}}] }
> 2013-08-05 16:58:45,762 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
SeqA 2-790: Sending Seq 2-790:  { Ans: , MgmtId: 7280707764394, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-08-05 16:58:46,685 DEBUG [network.resource.NetscalerResource] (DirectAgent-169:null)
Netscaler load balancer 10.147.60.26 successfully executed IPAssocCommand to remove IP com.cloud.agent.api.to.IpAddressTO@1e6c629
> 2013-08-05 16:58:46,685 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-169:null)
Seq 8-1242890258: Response Received:
> 2013-08-05 16:58:46,685 DEBUG [agent.transport.Request] (DirectAgent-169:null) Seq 8-1242890258:
Processing:  { Ans: , MgmtId: 7280707764394, via: 8, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.IpAssocAnswer":{"results":["null
- success"],"result":true,"wait":0}}] }
> 2013-08-05 16:58:46,686 DEBUG [agent.transport.Request] (Network-Scavenger-1:null) Seq
8-1242890258: Received:  { Ans: , MgmtId: 7280707764394, via: 8, Ver: v1, Flags: 10, { IpAssocAnswer
} }
> 2013-08-05 16:58:46,800 DEBUG [cloud.network.ExternalLoadBalancerDeviceManagerImpl] (Network-Scavenger-1:null)
External load balancer has shut down the guest network for account dom1Acc2(id = 4) with VLAN
tag 909
> 2013-08-05 16:58:46,804 DEBUG [cloud.network.NetworkManagerImpl] (Network-Scavenger-1:null)
Sending network shutdown to VirtualRouter
> 2013-08-05 16:58:46,808 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Network-Scavenger-1:null)
Stopping router VM[DomainRouter|r-14-VM]
> 2013-08-05 16:58:46,817 DEBUG [cloud.capacity.CapacityManagerImpl] (Network-Scavenger-1:null)
VM state transitted from :Running to Stopping with event: StopRequestedvm's original host
id: 1 new host id: 1 host id before state transition: 1
> 2013-08-05 16:58:46,945 DEBUG [agent.transport.Request] (AgentManager-Handler-1:null)
Seq 1-1321009498: Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-14-VM","bytesSent":0,"bytesReceived":0,"result":true,"details":"","wait":0}}]
}
> Attaching all the required logs along with db dump to the bug.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message