cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ja kub <jjaku...@gmail.com>
Subject Re: problem with release test procedure
Date Sun, 19 Jan 2014 21:41:31 GMT
I am running the management server not from within DevCloud, but outside of
virtualbox - on my linux.
Should it be run from virtualbox  from 192.168.56.10 ?

in webadmin I see:

   -  Secondary Storage VM

   VM (name: s-71-VMid: 71) stopped unexpectedly on host id:1availability
   zone id:1pod id:1

   Sun19 Jan 2014 20:56:01 GMT


   -  Zone: DevCloud0
    100%
    Management IP Addresses
    21 / 21

Now I made (hope cleaned whole database and filled from scratch):

mvn -P developer -pl developer,tools/devcloud -Ddeploydb
mvn -pl :cloud-client-ui jetty:run

mvn -P developer -pl tools/devcloud -Ddeploysvr

And I get other errors:
In console is

   -
      -  Zone: DevCloud0
       100%
       Management IP Addresses
       21 / 21    (when I was copying logs it was 19/21)

Any idea what can I be doing wrong ?


in logs is:

INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) No
stopped console proxy is available, need to allocate a new console proxy
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 1

...

com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
....
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-1:) Seq 1-490209293:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
....
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
...
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 1
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-1-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
...

full log:


INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.agent.manager.allocator.impl.FirstFitRoutingAllocator_EnhancerByCloudStack_6c462da3
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.baremetal.networkservice.BaremetalUserdataElement_EnhancerByCloudStack_60b19f64
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.server.auth.LDAPUserAuthenticator_EnhancerByCloudStack_d7f99c9a
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.agent.manager.allocator.impl.UserConcentratedAllocator_EnhancerByCloudStack_d717c8dd
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.network.guru.OvsGuestNetworkGuru_EnhancerByCloudStack_7bd035b6
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.template.HypervisorTemplateAdapter_EnhancerByCloudStack_4b9b8aa
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.network.guru.StorageNetworkGuru_EnhancerByCloudStack_e6ea3acd
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.network.element.MidoNetElement_EnhancerByCloudStack_655a2d48
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
org.apache.cloudstack.storage.allocator.ClusterScopeStoragePoolAllocator_EnhancerByCloudStack_fc0db8cb
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
org.apache.cloudstack.storage.allocator.GarbageCollectingStoragePoolAllocator_EnhancerByCloudStack_986e4950
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
org.apache.cloudstack.acl.StaticRoleBasedAPIAccessChecker_EnhancerByCloudStack_7000d2bf
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.ha.KVMFencer_EnhancerByCloudStack_53a4438e
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.network.element.VirtualRouterElement_EnhancerByCloudStack_f704659a
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.ha.XenServerFencer_EnhancerByCloudStack_19852100
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.server.auth.MD5UserAuthenticator_EnhancerByCloudStack_6ea0343d
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.storage.secondary.SecondaryStorageVmDefaultAllocator_EnhancerByCloudStack_5850616d
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.network.element.NiciraNvpElement_EnhancerByCloudStack_a27ffc76
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.hypervisor.XenServerGuru_EnhancerByCloudStack_9b3fcd5e
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.ha.XenServerInvestigator_EnhancerByCloudStack_5119bbc8
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.network.element.ElasticLoadBalancerElement_EnhancerByCloudStack_aaac5195
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.network.guru.NiciraNvpGuestNetworkGuru_EnhancerByCloudStack_f8c6dba2
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.baremetal.networkservice.BaremetalPxeElement_EnhancerByCloudStack_4638ff46
INFO  [utils.component.ComponentContext] (Timer-2:) Starting
com.cloud.server.ManagementServerImpl_EnhancerByCloudStack_aea9ab9d
INFO  [cloud.server.ManagementServerImpl] (Timer-2:) Startup CloudStack
management server...
INFO  [cloud.server.ManagementServerImpl] (Timer-2:) Admin user enabled
INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:) We are
good, no orphan management server msid in host table is found
INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:) No inactive
management server node found
WARN  [apache.cloudstack.alerts] (Cluster-Notification-1:)  alertType:: 14
// dataCenterId:: 0 // podId:: 0 // clusterId:: null // message::
Management server node 127.0.0.1 is up
WARN  [cloud.cluster.ClusterManagerImpl] (Cluster-Notification-1:)
Notifying management server join event took 25 ms
INFO  [cloud.configuration.ConfigurationManagerImpl]
(1010744083@qtp-35133715-9:) adding a new subnet to the network 204
INFO  [cloud.resource.ResourceManagerImpl] (1010744083@qtp-35133715-9:)
Trying to add a new host at http://192.168.56.10/ in data center 1
INFO  [xen.discoverer.XcpServerDiscoverer] (1010744083@qtp-35133715-9:)
Found host devcloud ip=192.168.56.10 product version=1.6.0
INFO  [xen.resource.CitrixResourceBase] (1010744083@qtp-35133715-9:)
Private Network is Pool-wide network associated with eth0 for host
192.168.56.10
INFO  [xen.resource.CitrixResourceBase] (1010744083@qtp-35133715-9:) Guest
Network is Pool-wide network associated with eth0 for host 192.168.56.10
INFO  [xen.resource.CitrixResourceBase] (1010744083@qtp-35133715-9:) Public
Network is Pool-wide network associated with eth0 for host 192.168.56.10
INFO  [xen.discoverer.XcpServerDiscoverer] (1010744083@qtp-35133715-9:)
Host: devcloud connected with hypervisor type: XenServer. Checking CIDR...
INFO  [agent.manager.DirectAgentAttache] (1010744083@qtp-35133715-9:)
StartupAnswer received 1 Interval = 60
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-1:) Host 192.168.56.10
OpaqueRef:0d19f3fa-7f04-543a-d906-aaf478fc574c: Host 192.168.56.10 is
already setup.
INFO  [network.security.SecurityGroupListener] (1010744083@qtp-35133715-9:)
Received a host startup notification
INFO  [network.security.SecurityGroupListener] (1010744083@qtp-35133715-9:)
Scheduled network rules cleanup, interval=2057
INFO  [network.security.SecurityGroupListener] (1010744083@qtp-35133715-9:)
Received a host startup notification
INFO  [cloud.resource.ResourceManagerImpl] (1010744083@qtp-35133715-9:)
server resources successfully discovered by XCP Agent
INFO  [datastore.lifecycle.CloudStackImageStoreLifeCycleImpl]
(1010744083@qtp-35133715-9:) Trying to add a new data store at nfs://
192.168.56.10/opt/storage/secondary to data center 1
INFO  [cloud.configuration.ConfigurationManagerImpl]
(1010744083@qtp-35133715-9:) No storage traffic type was specified by
admin, create default storage traffic on physical network 200 with same
configure of management traffic type
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) No
stopped console proxy is available, need to allocate a new console proxy
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 1
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-6:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-6:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-6:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 1
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref1 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-1:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-1:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-1:) Seq 1-490209293:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 1
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-1-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-11:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 1, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 1
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 2
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-7:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-7:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-7:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 2
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref2 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[1|vm=1|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-5:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-1-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 1, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 1 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 2
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-8:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-8:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-8:) Seq 1-490209297:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 3
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-17:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-17:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-17:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 3
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref3 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-18:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-18:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-18:) Seq 1-490209301:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 3
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-3-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-19:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 3, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 3
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 4
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-10:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-10:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-10:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 4
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref4 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[3|vm=3|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-2:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-3-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 3, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 3 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 4
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-20:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-20:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-20:) Seq 1-490209305:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 5
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-24:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-24:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-24:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 5
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref5 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-16:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-16:) Seq 1-490209309:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 5
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-4-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-17:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 4, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 4
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 6
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-18:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-18:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-18:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 6
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref6 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[4|vm=4|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-19:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-4-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 4, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 4 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 6
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-29:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-29:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-29:) Seq 1-490209313:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 7
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-11:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-11:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-11:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 7
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref7 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-7:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-7:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-7:) Seq 1-490209317:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 7
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-5-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-30:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 5, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 5
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 8
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-31:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-31:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-31:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 8
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref8 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[5|vm=5|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-20:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-5-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 5, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 5 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 8
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-34:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-34:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-34:) Seq 1-490209321:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 9
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-37:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-37:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-37:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 9
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref9 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[6|vm=6|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-38:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-6-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 6, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 9
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-39:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-39:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-39:) Seq 1-490209326:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 6
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 10
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-25:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-25:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-25:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 10
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref10 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[6|vm=6|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-4:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-6-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 6, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 6 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 10
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-15:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-15:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-15:) Seq 1-490209329:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 11
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-42:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-42:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-42:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 11
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref11 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-43:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-43:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-43:) Seq 1-490209333:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 11
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-7-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-44:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 7, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 7
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 12
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-45:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-45:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-45:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 12
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref12 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[7|vm=7|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-46:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-7-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 7, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 7 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 12
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-30:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-30:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-30:) Seq 1-490209337:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 13
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-3:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-3:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-3:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 13
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref13 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-1:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-1:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-1:) Seq 1-490209341:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 13
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-8-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-14:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 8, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 8
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 14
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-10:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-10:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-10:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 14
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref14 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[8|vm=8|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-33:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-8-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 8, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 8 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 14
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-34:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-34:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-34:) Seq 1-490209345:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 15
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-37:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-37:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-37:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 15
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref15 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-38:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-38:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-38:) Seq 1-490209349:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 15
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-9-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-39:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 9, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 9
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 16
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-25:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-25:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-25:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 16
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref16 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[9|vm=9|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-4:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-9-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 9, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 9 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 16
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-8:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-8:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-8:) Seq 1-490209353:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 17
WARN  [apache.cloudstack.alerts] (CapacityChecker:)  alertType:: 5 //
dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: System
Alert: Number of unallocated private IPs is low in pod test00 of
availability zone DevCloud0
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-60:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-60:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-60:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 17
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref17 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[10|vm=10|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-61:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-10-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 10, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 17
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-15:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-15:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-15:) Seq 1-490209358:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 10
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 18
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-26:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-26:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-26:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 18
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref18 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[10|vm=10|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-12:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-10-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 10, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 10 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) Unable to
acquire lock on VMTemplateStoragePool 18
ERROR [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Failed to
start instance VM[ConsoleProxy|v-2-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-66:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-66:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-66:) Seq 1-490209361:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 19
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-69:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-69:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-69:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 19
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref19 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-27:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-27:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-27:) Seq 1-490209365:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Unable to acquire
lock on VMTemplateStoragePool 19
ERROR [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Failed to start
instance VM[SecondaryStorageVm|s-11-VM]
java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-29:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:999)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:428)
    at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:575)
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2577)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    ... 20 more
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 11, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 11
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 20
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Insufficient
capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
    at
com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121)
    at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2166)
    at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-40:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-40:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-40:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 20
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref20 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[11|vm=11|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-5:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-5:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-5:) Seq 1-490209368:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-70:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-11-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 11, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 11 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Insufficient
capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
    at
com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121)
    at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2166)
    at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 21
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-74:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-74:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-74:) Seq 1-490209372:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-75:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-75:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-75:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 21
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref21 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[12|vm=12|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-76:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-12-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 12, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 12
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) lock is acquired
for VMTemplateStoragePool 22
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Insufficient
capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
    at
com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121)
    at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2166)
    at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-77:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-77:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-77:) Seq 1-490209375:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-78:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-78:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-78:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) releasing lock for
VMTemplateStoragePool 22
WARN  [utils.db.Merovingian2] (secstorage-1:) Was unable to find lock for
the key template_spool_ref22 and thread id 1477320197
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[12|vm=12|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-79:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-12-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 12, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 12 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
running ssvm is found, so command will be sent to LocalHostEndPoint
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) No
stopped secondary storage vm is available, need to allocate a new secondary
storage vm
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) lock is acquired
for VMTemplateStoragePool 23
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Insufficient
capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
    at
com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121)
    at
com.cloud.network.guru.StorageNetworkGuru.reserve(StorageNetworkGuru.java:120)
    at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2166)
    at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-48:) kill_copy_process
failed
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-48:) can not
create vdi in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
WARN  [xen.resource.XenServerStorageProcessor] (DirectAgent-48:) Catch
Exception com.cloud.utils.exception.CloudRuntimeException for template +
due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi
in sr 9f3f9262-3f77-09cc-2df7-0d8475676260
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
    at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)
    at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.volume.VolumeServiceImpl] (consoleproxy-1:) releasing lock
for VMTemplateStoragePool 23
WARN  [utils.db.Merovingian2] (consoleproxy-1:) Was unable to find lock for
the key template_spool_ref23 and thread id 765519295
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Unable to
contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1]
is unreachable: Unable to create Vol[2|vm=2|ROOT]:Catch Exception
com.cloud.utils.exception.CloudRuntimeException for template +  due to
com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr
9f3f9262-3f77-09cc-2df7-0d8475676260
    at
com.cloud.storage.VolumeManagerImpl.recreateVolume(VolumeManagerImpl.java:2590)
    at
com.cloud.storage.VolumeManagerImpl.prepare(VolumeManagerImpl.java:2641)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:888)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-50:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-13-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 13, will recycle it and start a new one
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-51:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-51:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-51:) Seq 1-490209382:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
No running secondary storage vms found in datacenter id=1, starting one
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) Found
a stopped secondary storage vm, bring it up to running pool. secStorageVm
vm id : 13
INFO  [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found
a stopped console proxy, bring it up to running pool. proxy vm id : 2
INFO  [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Insufficient
capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
    at
com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121)
    at
com.cloud.network.guru.StorageNetworkGuru.reserve(StorageNetworkGuru.java:120)
    at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2166)
    at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:) Insufficient
capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
    at
com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:121)
    at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2166)
    at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-82:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-13-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
    at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
    at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
INFO  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:)
Unable to start secondary storage vm for standby capacity, secStorageVm vm
Id : 13, will recycle it and start a new one
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Expunge volume
with no data store specified
INFO  [storage.volume.VolumeServiceImpl] (secstorage-1:) Volume 13 is not
referred anywhere, remove it from volumes table
INFO  [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:)
Primary secondary storage is not even started, wait until next turn
INFO  [xen.resource.CitrixResourceBase] (DirectAgent-83:) VM does not exist
on XenServer9d4c9db8-32f7-25c3-0435-eab4bf3adcea
WARN  [xen.resource.CitrixResourceBase] (DirectAgent-83:) callHostPlugin
failed for cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
WARN  [agent.manager.DirectAgentAttache] (DirectAgent-83:) Seq 1-490209384:
Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for
cmd: setDNATRule with args add: false,  due to
UNKNOWN_XENAPI_PLUGIN_FUNCTIONsetDNATRule
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:172)
    at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:503)
    at
com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:143)
    at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
WARN  [cloud.vm.SystemVmLoadScanner] (consoleproxy-1:) Unexpected exception
com.cloud.agent.api.Answer cannot be cast to com.cloud.agent.api.StopAnswer
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.StopAnswer
    at
com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1065)
    at
com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1109)
    at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:1009)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
    at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
    at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
    at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
    at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
    at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
    at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)




On Fri, Jan 17, 2014 at 8:14 PM, Amogh Vasekar <amogh.vasekar@citrix.com>wrote:

> Hi,
> Is the SSVM in up when checked from the UI? You may need to change the
> "host" config in global parameters to 192.168.56.10 (assuming you are
> running the management server from within DevCloud).
>
> HTH,
> Amogh
>
> On 1/17/14 7:47 AM, "Ja kub" <jjakub83@gmail.com> wrote:
>
> >Hi,
> >
> >
> >Has anybody recently tested creating instance with virtualbox image ?
> >Was it successful ?
> >
> >I am a complete newbie,
> >
> >I try to do
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+test+proced
> >ure
> >
> >and
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/DevCloud#DevCloud-S
> >ettingup%28VirtualBox%29
> >
> >I donwloaded and imported to virtualbox devcloud2.ova image.
> >
> >I can ssh to it as root,
> >
> >after mvn -P developer -pl tools/devcloud -Ddeploysvr
> >it is visible in hosts,
> >
> >I can see two templates in templates tab, but I cannot create new
> >instance,
> >because I see no template in create new instance flow.
> >
> >after jetty start at end of logs I can see:
> >
> >INFO  [utils.component.ComponentContext] (Timer-2:) Starting
> >com.cloud.baremetal.networkservice.BaremetalPxeElement_EnhancerByCloudStac
> >k_5e4410f
> >INFO  [utils.component.ComponentContext] (Timer-2:) Starting
> >com.cloud.server.ManagementServerImpl_EnhancerByCloudStack_6e54ed66
> >INFO  [cloud.server.ManagementServerImpl] (Timer-2:) Startup CloudStack
> >management server...
> >INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:) We are
> >good, no orphan management server msid in host table is found
> >INFO  [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:) No
> >inactive
> >management server node found
> >WARN  [apache.cloudstack.alerts] (Cluster-Notification-1:)  alertType:: 14
> >// dataCenterId:: 0 // podId:: 0 // clusterId:: null // message::
> >Management server node 127.0.0.1 is up
> >WARN  [cloud.cluster.ClusterManagerImpl] (Cluster-Notification-1:)
> >Notifying management server join event took 35 ms
> >INFO  [xen.resource.CitrixResourceBase] (AgentTaskPool-1:) Private Network
> >is Pool-wide network associated with eth0 for host 192.168.56.10
> >INFO  [xen.resource.CitrixResourceBase] (AgentTaskPool-1:) Guest Network
> >is
> >Pool-wide network associated with eth0 for host 192.168.56.10
> >INFO  [xen.resource.CitrixResourceBase] (AgentTaskPool-1:) Public Network
> >is Pool-wide network associated with eth0 for host 192.168.56.10
> >INFO  [xen.discoverer.XcpServerDiscoverer] (AgentTaskPool-1:) Host:
> >devcloud connected with hypervisor type: XenServer. Checking CIDR...
> >INFO  [agent.manager.DirectAgentAttache] (AgentTaskPool-1:) StartupAnswer
> >received 1 Interval = 60
> >INFO  [xen.resource.CitrixResourceBase] (DirectAgent-1:) Host
> >192.168.56.10
> >OpaqueRef:0d19f3fa-7f04-543a-d906-aaf478fc574c: Host 192.168.56.10 is
> >already setup.
> >INFO  [network.security.SecurityGroupListener] (AgentTaskPool-1:) Received
> >a host startup notification
> >INFO  [network.security.SecurityGroupListener] (AgentTaskPool-1:)
> >Scheduled
> >network rules cleanup, interval=2406
> >INFO  [network.security.SecurityGroupListener] (AgentTaskPool-1:) Received
> >a host startup notification
> >INFO  [storage.image.TemplateServiceImpl] (AgentTaskPool-1:) Downloading
> >builtin template tiny Linux to data center: 1
> >INFO  [storage.endpoint.DefaultEndPointSelector] (AgentTaskPool-1:) No
> >running ssvm is found, so command will be sent to LocalHostEndPoint
> >INFO  [storage.endpoint.DefaultEndPointSelector] (StatsCollector-3:) No
> >running ssvm is found, so command will be sent to LocalHostEndPoint
> >WARN  [apache.cloudstack.alerts] (CapacityChecker:)  alertType:: 5 //
> >dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: System
> >Alert: Number of unallocated private IPs is low in pod test00 of
> >availability zone DevCloud0
> >
> >
> >BR
> >Jakub
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message