cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailaja Mada (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (CLOUDSTACK-4090) [Upgrade] System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is created after upgrade
Date Thu, 22 Aug 2013 06:36:52 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sailaja Mada closed CLOUDSTACK-4090.
------------------------------------


This issue is not observed with 305 Patch A -> 307 -> 4.2 path & 307 -> 4.2 path.

Hence closing the bug. 
                
> [Upgrade] System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is created
after upgrade 
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4090
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4090
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: VMware
>    Affects Versions: 4.2.0
>            Reporter: Sailaja Mada
>            Assignee: frank zhang
>            Priority: Critical
>             Fix For: 4.2.0
>
>         Attachments: apilog.log, Bforeupgraedecloud-backup.dmp, postupgradecloud-backup.dmp
>
>
> Steps:
> 1. Upgraded from 305 Patch A to 4.2 with Xenserver Zone
> 2. Enabled Nexus global config value 
> 3. Configure one more Adv zone with VMWARE cluster with Nexus 1000v Switch enabled 
> 4. Seeded secondary storage with new 4.2 vMWARE template (vh7)
> Physical network 1 : 
> Mgmt Traffic label name  - vSwitch0,,vmwaresvs 
> Physical network 1 : 
> Public & Guest Traffic label name  - nexuspp91,,nexusdvs
> Observation:System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is
created after upgrade 
> After upgrade I have enabled Nexus vSwtich and configured zone with Nexus cluster.  Now
system VM’s are coming up now : I I am using vh7 template) - 
> 2013-08-05 21:01:52,084 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null)
Found 0 networks to update RvR status.
> 2013-08-05 21:01:52,091 DEBUG [storage.motion.AncientDataMotionStrategy] (consoleproxy-1:null)
copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
> 2013-08-05 21:01:52,174 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239:
Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
}
> 2013-08-05 21:01:52,175 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239:
Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM
Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}]
}
> 2013-08-05 21:01:52,186 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null)
Seq 8-268894239: Executing request
> 2013-08-05 21:01:52,318 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null)
Found 5 routers to update status.
> 2013-08-05 21:01:52,323 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null)
Found 0 networks to update RvR status.
> 2013-08-05 21:01:52,330 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Template routing-8 is not setup yet, setup template from secondary storage with uuid name:
431f50e387b738feabae43f819e69bbf
> 2013-08-05 21:01:52,345 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/,
templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
> 2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5e4f2a84
> 2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.d778a15
> 2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.6b16fc2f
> 2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.8558063
> 2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5b202dbe
> 2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.20c1ddf
> 2013-08-05 21:01:52,382 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.3a381db8
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5223dd26
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.2c371333
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5ad80648
> 2013-08-05 21:01:52,383 WARN  [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create a mount point
> 2013-08-05 21:01:52,384 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20)
Unable to create mount point for nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/
> 2013-08-05 21:01:52,384 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Secondary storage mount point: /mnt/sec
> 2013-08-05 21:01:52,388 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Executing command: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:52,388 DEBUG [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Executing: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:53,873 WARN  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Exception: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"):
java.io.IOException: error=2, No such file or directory
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
>         at com.cloud.utils.script.Script.execute(Script.java:183)
>         at com.cloud.utils.script.Script.execute(Script.java:161)
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:145)
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
>         at java.lang.ProcessImpl.start(ProcessImpl.java:81)
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
>         ... 16 more
> 2013-08-05 21:01:53,878 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:53,879 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20)
Unable to execute PrimaryStorageDownloadCommand due to exception
> java.lang.Exception: Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:149)
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-05 21:01:53,880 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null)
Seq 8-268894239: Response Received:
> 2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (DirectAgent-204:null) Seq 8-268894239:
Processing:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Unable
to execute PrimaryStorageDownloadCommand due to exception","wait":0}}] }
> 2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239:
Received:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, { CopyCmdAnswer } }
> 2013-08-05 21:01:53,906 INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:null)
releasing lock for VMTemplateStoragePool 11
> 2013-08-05 21:01:53,906 WARN  [utils.db.Merovingian2] (consoleproxy-1:null) Was unable
to find lock for the key template_spool_ref11 and thread id 144163998
> 2013-08-05 21:01:53,907 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null)
Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to
exception
> 2013-08-05 21:01:53,922 INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null)
Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:202] is unreachable:
Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to
exception
>         at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2496)
>         at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
>         at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
>         at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:617)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1665)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
>         at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
>         at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
>         at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
>         at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-05 21:01:53,953 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null)
Cleaning up resources for the vm VM[ConsoleProxy|v-21-VM] in Starting state
> 2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240:
Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}]
}
> 2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240:
Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}]
}
> 2013-08-05 21:01:53,990 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null)
Seq 8-268894240: Executing request
> 2013-08-05 21:01:53,991 INFO  [vmware.resource.VmwareResource] (DirectAgent-163:10.102.192.20)
Executing resource StopCommand: {"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}
> mysql> select * from physical_network_traffic_types;
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> | id | uuid                                 | physical_network_id | traffic_type | xen_network_label
| kvm_network_label | vmware_network_label | simulator_network_label | ovm_network_label |
vlan | lxc_network_label |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> |  1 | bb74693a-b32a-499d-94b6-397e871f96c7 |                 200 | Management   | NULL
             | NULL              | NULL                 | NULL                    | NULL 
            | NULL | cloudbr0          |
> |  2 | 58a43a6a-f4f1-4936-b140-a3429d7b95ee |                 200 | Public       | NULL
             | NULL              | NULL                 | NULL                    | NULL 
            | NULL | cloudbr0          |
> |  3 | 22e56281-dfaa-4bb0-aedc-d4866d4c118a |                 200 | Guest        | NULL
             | NULL              | NULL                 | NULL                    | NULL 
            | NULL | cloudbr0          |
> |  4 | dbe80dd9-c70c-4b38-bfb7-4e8bfeca0768 |                 200 | Storage      | NULL
             | NULL              | NULL                 | NULL                    | NULL 
            | NULL | cloudbr0          |
> |  5 | 63c8710b-b644-4a6d-bb58-5886176c0688 |                 202 | Public       | NULL
             | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL 
            | NULL | cloudbr0          |
> |  6 | 6ba2196e-fc1a-4822-ba4d-ae2a8af2df53 |                 202 | Guest        | NULL
             | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL 
            | NULL | cloudbr0          |
> |  7 | 4397e50a-4d30-4f19-b75c-ce4f3d070cb8 |                 201 | Management   | NULL
             | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL 
            | NULL | cloudbr0          |
> |  8 | 261b29ef-521b-4a6a-8b91-2328431f6bb5 |                 201 | Storage      | NULL
             | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL 
            | NULL | cloudbr0          |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> 8 rows in set (0.00 sec)
> mysql>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message