cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailaja Mada (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-4090) System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is created after upgrade
Date Mon, 05 Aug 2013 16:52:48 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sailaja Mada updated CLOUDSTACK-4090:
-------------------------------------

    Description: 
Steps:

1. Upgraded from 305 Patch A to 4.2 with Xenserver Zone
2. Enabled Nexus global config value 
3. Configure one more Adv zone with VMWARE cluster with Nexus 1000v Switch enabled 
4. Seeded secondary storage with new 4.2 vMWARE template (vh7)

Physical network 1 : 
Mgmt Traffic label name  - vSwitch0,,vmwaresvs 

Physical network 1 : 
Public & Guest Traffic label name  - nexuspp91,,nexusdvs

Observation:System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is created after upgrade 

After upgrade I have enabled Nexus vSwtich and configured zone with Nexus cluster.  Now system VM’s are coming up now : I I am using vh7 template) - 


2013-08-05 21:01:52,084 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-08-05 21:01:52,091 DEBUG [storage.motion.AncientDataMotionStrategy] (consoleproxy-1:null) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2013-08-05 21:01:52,174 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}] }
2013-08-05 21:01:52,175 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}] }
2013-08-05 21:01:52,186 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null) Seq 8-268894239: Executing request
2013-08-05 21:01:52,318 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 5 routers to update status.
2013-08-05 21:01:52,323 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-08-05 21:01:52,330 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Template routing-8 is not setup yet, setup template from secondary storage with uuid name: 431f50e387b738feabae43f819e69bbf
2013-08-05 21:01:52,345 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/, templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5e4f2a84
2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.d778a15
2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.6b16fc2f
2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.8558063
2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5b202dbe
2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.20c1ddf
2013-08-05 21:01:52,382 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.3a381db8
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5223dd26
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.2c371333
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5ad80648
2013-08-05 21:01:52,383 WARN  [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create a mount point
2013-08-05 21:01:52,384 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount point for nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/
2013-08-05 21:01:52,384 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Secondary storage mount point: /mnt/sec
2013-08-05 21:01:52,388 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing command: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:52,388 DEBUG [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:53,873 WARN  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Exception: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
        at com.cloud.utils.script.Script.execute(Script.java:183)
        at com.cloud.utils.script.Script.execute(Script.java:161)
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:145)
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
        at java.lang.ProcessImpl.start(ProcessImpl.java:81)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
        ... 16 more
2013-08-05 21:01:53,878 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:53,879 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Unable to execute PrimaryStorageDownloadCommand due to exception
java.lang.Exception: Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:149)
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-08-05 21:01:53,880 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null) Seq 8-268894239: Response Received:
2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (DirectAgent-204:null) Seq 8-268894239: Processing:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Unable to execute PrimaryStorageDownloadCommand due to exception","wait":0}}] }
2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Received:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-08-05 21:01:53,906 INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:null) releasing lock for VMTemplateStoragePool 11
2013-08-05 21:01:53,906 WARN  [utils.db.Merovingian2] (consoleproxy-1:null) Was unable to find lock for the key template_spool_ref11 and thread id 144163998
2013-08-05 21:01:53,907 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to exception
2013-08-05 21:01:53,922 INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:202] is unreachable: Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to exception
        at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2496)
        at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
        at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
        at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:617)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1665)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
        at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
        at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
        at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
        at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-08-05 21:01:53,953 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Cleaning up resources for the vm VM[ConsoleProxy|v-21-VM] in Starting state
2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240: Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}] }
2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240: Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}] }
2013-08-05 21:01:53,990 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq 8-268894240: Executing request
2013-08-05 21:01:53,991 INFO  [vmware.resource.VmwareResource] (DirectAgent-163:10.102.192.20) Executing resource StopCommand: {"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}


mysql> select * from physical_network_traffic_types;
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
| id | uuid                                 | physical_network_id | traffic_type | xen_network_label | kvm_network_label | vmware_network_label | simulator_network_label | ovm_network_label | vlan | lxc_network_label |
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
|  1 | bb74693a-b32a-499d-94b6-397e871f96c7 |                 200 | Management   | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  2 | 58a43a6a-f4f1-4936-b140-a3429d7b95ee |                 200 | Public       | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  3 | 22e56281-dfaa-4bb0-aedc-d4866d4c118a |                 200 | Guest        | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  4 | dbe80dd9-c70c-4b38-bfb7-4e8bfeca0768 |                 200 | Storage      | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  5 | 63c8710b-b644-4a6d-bb58-5886176c0688 |                 202 | Public       | NULL              | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL              | NULL | cloudbr0          |
|  6 | 6ba2196e-fc1a-4822-ba4d-ae2a8af2df53 |                 202 | Guest        | NULL              | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL              | NULL | cloudbr0          |
|  7 | 4397e50a-4d30-4f19-b75c-ce4f3d070cb8 |                 201 | Management   | NULL              | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL              | NULL | cloudbr0          |
|  8 | 261b29ef-521b-4a6a-8b91-2328431f6bb5 |                 201 | Storage      | NULL              | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL              | NULL | cloudbr0          |
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
8 rows in set (0.00 sec)

mysql>





  was:
Steps:

1. Upgraded from 305 Patch A to 4.2 with Xenserver Zone
2. Enabled Nexus global config value 
3. Configure one more Adv zone with VMWARE cluster with Nexus 1000v Switch enabled 

Physical network 1 : 
Mgmt Traffic label name  - vSwitch0,,vmwaresvs 

Physical network 1 : 
Public & Guest Traffic label name  - nexuspp91,,nexusdvs

Observation:System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is created after upgrade 

After upgrade I have enabled Nexus vSwtich and configured zone with Nexus cluster.  Now system VM’s are coming up now : I I am using vh7 template) - 


2013-08-05 21:01:52,084 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-08-05 21:01:52,091 DEBUG [storage.motion.AncientDataMotionStrategy] (consoleproxy-1:null) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2013-08-05 21:01:52,174 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}] }
2013-08-05 21:01:52,175 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}] }
2013-08-05 21:01:52,186 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null) Seq 8-268894239: Executing request
2013-08-05 21:01:52,318 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 5 routers to update status.
2013-08-05 21:01:52,323 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-08-05 21:01:52,330 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Template routing-8 is not setup yet, setup template from secondary storage with uuid name: 431f50e387b738feabae43f819e69bbf
2013-08-05 21:01:52,345 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/, templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5e4f2a84
2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.d778a15
2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.6b16fc2f
2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.8558063
2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5b202dbe
2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.20c1ddf
2013-08-05 21:01:52,382 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.3a381db8
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5223dd26
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.2c371333
2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5ad80648
2013-08-05 21:01:52,383 WARN  [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create a mount point
2013-08-05 21:01:52,384 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount point for nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/
2013-08-05 21:01:52,384 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Secondary storage mount point: /mnt/sec
2013-08-05 21:01:52,388 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing command: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:52,388 DEBUG [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:53,873 WARN  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Exception: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
        at com.cloud.utils.script.Script.execute(Script.java:183)
        at com.cloud.utils.script.Script.execute(Script.java:161)
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:145)
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
        at java.lang.ProcessImpl.start(ProcessImpl.java:81)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
        ... 16 more
2013-08-05 21:01:53,878 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
2013-08-05 21:01:53,879 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Unable to execute PrimaryStorageDownloadCommand due to exception
java.lang.Exception: Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:149)
        at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
        at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
        at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-08-05 21:01:53,880 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null) Seq 8-268894239: Response Received:
2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (DirectAgent-204:null) Seq 8-268894239: Processing:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Unable to execute PrimaryStorageDownloadCommand due to exception","wait":0}}] }
2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Received:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-08-05 21:01:53,906 INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:null) releasing lock for VMTemplateStoragePool 11
2013-08-05 21:01:53,906 WARN  [utils.db.Merovingian2] (consoleproxy-1:null) Was unable to find lock for the key template_spool_ref11 and thread id 144163998
2013-08-05 21:01:53,907 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to exception
2013-08-05 21:01:53,922 INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:202] is unreachable: Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to exception
        at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2496)
        at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
        at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
        at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
        at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:617)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1665)
        at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
        at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
        at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
        at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
        at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2013-08-05 21:01:53,953 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Cleaning up resources for the vm VM[ConsoleProxy|v-21-VM] in Starting state
2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240: Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}] }
2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240: Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}] }
2013-08-05 21:01:53,990 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq 8-268894240: Executing request
2013-08-05 21:01:53,991 INFO  [vmware.resource.VmwareResource] (DirectAgent-163:10.102.192.20) Executing resource StopCommand: {"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}


mysql> select * from physical_network_traffic_types;
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
| id | uuid                                 | physical_network_id | traffic_type | xen_network_label | kvm_network_label | vmware_network_label | simulator_network_label | ovm_network_label | vlan | lxc_network_label |
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
|  1 | bb74693a-b32a-499d-94b6-397e871f96c7 |                 200 | Management   | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  2 | 58a43a6a-f4f1-4936-b140-a3429d7b95ee |                 200 | Public       | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  3 | 22e56281-dfaa-4bb0-aedc-d4866d4c118a |                 200 | Guest        | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  4 | dbe80dd9-c70c-4b38-bfb7-4e8bfeca0768 |                 200 | Storage      | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
|  5 | 63c8710b-b644-4a6d-bb58-5886176c0688 |                 202 | Public       | NULL              | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL              | NULL | cloudbr0          |
|  6 | 6ba2196e-fc1a-4822-ba4d-ae2a8af2df53 |                 202 | Guest        | NULL              | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL              | NULL | cloudbr0          |
|  7 | 4397e50a-4d30-4f19-b75c-ce4f3d070cb8 |                 201 | Management   | NULL              | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL              | NULL | cloudbr0          |
|  8 | 261b29ef-521b-4a6a-8b91-2328431f6bb5 |                 201 | Storage      | NULL              | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL              | NULL | cloudbr0          |
+----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
8 rows in set (0.00 sec)

mysql>





    
> System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is created after upgrade 
> ------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4090
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4090
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: VMware
>    Affects Versions: 4.2.0
>            Reporter: Sailaja Mada
>            Priority: Blocker
>             Fix For: 4.2.0
>
>         Attachments: apilog.log, Bforeupgraedecloud-backup.dmp, postupgradecloud-backup.dmp
>
>
> Steps:
> 1. Upgraded from 305 Patch A to 4.2 with Xenserver Zone
> 2. Enabled Nexus global config value 
> 3. Configure one more Adv zone with VMWARE cluster with Nexus 1000v Switch enabled 
> 4. Seeded secondary storage with new 4.2 vMWARE template (vh7)
> Physical network 1 : 
> Mgmt Traffic label name  - vSwitch0,,vmwaresvs 
> Physical network 1 : 
> Public & Guest Traffic label name  - nexuspp91,,nexusdvs
> Observation:System VM's are failed to start on a new zone(VMWARE Nexus Zone) which is created after upgrade 
> After upgrade I have enabled Nexus vSwtich and configured zone with Nexus cluster.  Now system VM’s are coming up now : I I am using vh7 template) - 
> 2013-08-05 21:01:52,084 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-08-05 21:01:52,091 DEBUG [storage.motion.AncientDataMotionStrategy] (consoleproxy-1:null) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
> 2013-08-05 21:01:52,174 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}] }
> 2013-08-05 21:01:52,175 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/8/","origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/","_role":"Image"}},"name":"routing-8","hypervisorType":"VMware"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova","uuid":"8","id":8,"format":"OVA","accountId":1,"checksum":"7137e453f950079ea2ba6feaafd939e8","hvm":false,"displayText":"SystemVM Template (vSphere)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"55e6d095-7e22-3f4a-b533-874a9fede8df","id":202,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vmwareps1","port":2049}},"name":"routing-8","hypervisorType":"VMware"}},"executeInSequence":false,"wait":10800}}] }
> 2013-08-05 21:01:52,186 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null) Seq 8-268894239: Executing request
> 2013-08-05 21:01:52,318 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 5 routers to update status.
> 2013-08-05 21:01:52,323 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-08-05 21:01:52,330 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Template routing-8 is not setup yet, setup template from secondary storage with uuid name: 431f50e387b738feabae43f819e69bbf
> 2013-08-05 21:01:52,345 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing copyTemplateFromSecondaryToPrimary. secondaryStorage: nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/, templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
> 2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5e4f2a84
> 2013-08-05 21:01:52,379 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.d778a15
> 2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.6b16fc2f
> 2013-08-05 21:01:52,380 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.8558063
> 2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5b202dbe
> 2013-08-05 21:01:52,381 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.20c1ddf
> 2013-08-05 21:01:52,382 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.3a381db8
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5223dd26
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.2c371333
> 2013-08-05 21:01:52,383 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount: /var/lib/cloud/mnt/VM/7674049379768.5ad80648
> 2013-08-05 21:01:52,383 WARN  [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create a mount point
> 2013-08-05 21:01:52,384 ERROR [vmware.manager.VmwareManagerImpl] (DirectAgent-204:10.102.192.20) Unable to create mount point for nfs://10.102.192.100/cpg_vol/sailaja/vmwaress1/
> 2013-08-05 21:01:52,384 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Secondary storage mount point: /mnt/sec
> 2013-08-05 21:01:52,388 INFO  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing command: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:52,388 DEBUG [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Executing: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:53,873 WARN  [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Exception: tar --no-same-owner -xf /mnt/sec/template/tmpl/1/8/routing-8.ova
> java.io.IOException: Cannot run program "tar" (in directory "/mnt/sec/template/tmpl/1/8"): java.io.IOException: error=2, No such file or directory
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
>         at com.cloud.utils.script.Script.execute(Script.java:183)
>         at com.cloud.utils.script.Script.execute(Script.java:161)
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:145)
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
>         at java.lang.ProcessImpl.start(ProcessImpl.java:81)
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
>         ... 16 more
> 2013-08-05 21:01:53,878 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
> 2013-08-05 21:01:53,879 ERROR [storage.resource.VmwareStorageProcessor] (DirectAgent-204:10.102.192.20) Unable to execute PrimaryStorageDownloadCommand due to exception
> java.lang.Exception: Unable to unpack snapshot OVA file at: /mnt/sec/template/tmpl/1/8/routing-8.ova
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateFromSecondaryToPrimary(VmwareStorageProcessor.java:149)
>         at com.cloud.storage.resource.VmwareStorageProcessor.copyTemplateToPrimaryStorage(VmwareStorageProcessor.java:241)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
>         at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
>         at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:562)
>         at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-05 21:01:53,880 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-204:null) Seq 8-268894239: Response Received:
> 2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (DirectAgent-204:null) Seq 8-268894239: Processing:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Unable to execute PrimaryStorageDownloadCommand due to exception","wait":0}}] }
> 2013-08-05 21:01:53,880 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894239: Received:  { Ans: , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 10, { CopyCmdAnswer } }
> 2013-08-05 21:01:53,906 INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:null) releasing lock for VMTemplateStoragePool 11
> 2013-08-05 21:01:53,906 WARN  [utils.db.Merovingian2] (consoleproxy-1:null) Was unable to find lock for the key template_spool_ref11 and thread id 144163998
> 2013-08-05 21:01:53,907 DEBUG [cloud.storage.VolumeManagerImpl] (consoleproxy-1:null) Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to exception
> 2013-08-05 21:01:53,922 INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Unable to contact resource.
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:202] is unreachable: Unable to create Vol[37|vm=21|ROOT]:Unable to execute PrimaryStorageDownloadCommand due to exception
>         at com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2496)
>         at com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2545)
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:934)
>         at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:624)
>         at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:617)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1665)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
>         at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
>         at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
>         at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
>         at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-05 21:01:53,953 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Cleaning up resources for the vm VM[ConsoleProxy|v-21-VM] in Starting state
> 2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240: Sending  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}] }
> 2013-08-05 21:01:53,984 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 8-268894240: Executing:  { Cmd , MgmtId: 7674049379768, via: 8, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}}] }
> 2013-08-05 21:01:53,990 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-163:null) Seq 8-268894240: Executing request
> 2013-08-05 21:01:53,991 INFO  [vmware.resource.VmwareResource] (DirectAgent-163:10.102.192.20) Executing resource StopCommand: {"isProxy":false,"executeInSequence":false,"vmName":"v-21-VM","wait":0}
> mysql> select * from physical_network_traffic_types;
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> | id | uuid                                 | physical_network_id | traffic_type | xen_network_label | kvm_network_label | vmware_network_label | simulator_network_label | ovm_network_label | vlan | lxc_network_label |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> |  1 | bb74693a-b32a-499d-94b6-397e871f96c7 |                 200 | Management   | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
> |  2 | 58a43a6a-f4f1-4936-b140-a3429d7b95ee |                 200 | Public       | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
> |  3 | 22e56281-dfaa-4bb0-aedc-d4866d4c118a |                 200 | Guest        | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
> |  4 | dbe80dd9-c70c-4b38-bfb7-4e8bfeca0768 |                 200 | Storage      | NULL              | NULL              | NULL                 | NULL                    | NULL              | NULL | cloudbr0          |
> |  5 | 63c8710b-b644-4a6d-bb58-5886176c0688 |                 202 | Public       | NULL              | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL              | NULL | cloudbr0          |
> |  6 | 6ba2196e-fc1a-4822-ba4d-ae2a8af2df53 |                 202 | Guest        | NULL              | NULL              | nexuspp91,,nexusdvs  | NULL                    | NULL              | NULL | cloudbr0          |
> |  7 | 4397e50a-4d30-4f19-b75c-ce4f3d070cb8 |                 201 | Management   | NULL              | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL              | NULL | cloudbr0          |
> |  8 | 261b29ef-521b-4a6a-8b91-2328431f6bb5 |                 201 | Storage      | NULL              | NULL              | vSwitch0,,vmwaresvs  | NULL                    | NULL              | NULL | cloudbr0          |
> +----+--------------------------------------+---------------------+--------------+-------------------+-------------------+----------------------+-------------------------+-------------------+------+-------------------+
> 8 rows in set (0.00 sec)
> mysql>

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message