cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sugandh S <s.suga...@rocketmail.com>
Subject Re: vm stuck in starting state, unable to delete it
Date Wed, 19 Mar 2014 10:52:59 GMT
Hi,

The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error".

Is there any other way to provide the log file?

Thanks,
Sugandh




On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sailaja.mada@citrix.com> wrote:
 
Hi,
 
Can you please send the complete log using PasteBin.     
 
Thanks,
Sailaja.M
 
From:Sugandh S [mailto:s.sugandh@rocketmail.com] 
Sent: 19 March 2014 15:30
To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S
Subject: Re: vm stuck in starting state, unable to delete it
 
Hi,
 
I just noticed that my domain router is also stuck in starting state and one of the vms I
created is now showing error state.
 
On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s.sugandh@rocketmail.com> wrote:
Hi,

> I have noticed VM in starting state when Template is getting  Copied  from Secondary
Storage to    > Primary Storage . 

It's been over 150 minutes and I don't think it should take this long to copy the template.

> size of the template and also value of global config parameter "wait" 

size of the iso is 700.29 MB and "wait" value is default "1800".

Sugandh




On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sailaja.mada@citrix.com> wrote:

Hi,

I have noticed VM in starting state when Template is getting  Copied  from Secondary Storage
to Primary Storage . 

VM gets deployed and will move to running state after this copy is completed. Can you please
share the size of the template and also value of global config parameter "wait" 

One reason could be Storage Server is slow and Copy operation is taking longer time.  It
would help not to time out if you increase the "wait" value . But you may have to wait for
the copy operation to complete to get the VM into running state.

Thanks,
Sailaja.M


-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com] 
Sent: 19 March 2014 15:01
To: Suresh Sadhu; users@cloudstack.apache.org
Subject: Re: vm stuck in starting state, unable to delete it

Here is another part of log

e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host
3 timed out after 3600
2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error
trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489660 to Host 3 timed out after 3600
2014-03-19 13:59:58,120 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null)
Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 13:59:58,120 WARN  [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901:
Timed out on null
2014-03-19 13:59:58,120 WARN  [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation
timed out: Commands 698
744901 to Host 4 timed out after 3600
2014-03-19 13:59:58,122 WARN  [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null)
Unable to obtain host 4 stati stics. 
2014-03-19 13:59:58,122 WARN  [cloud.server.StatsCollector] (StatsCollector-1:null) Received
invalid host stats for host
: 4
2014-03-19 14:00:07,507 WARN  [apache.cloudstack.alerts] (HA-2:null)  alertType:: 13 //
dataCenterId:: 0 // podId:: 0 //
 clusterId:: null // message:: No usage server process running
2014-03-19 14:00:22,973 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null)
Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:00:22,973 WARN  [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661:
Timed out on null
2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null)
Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands
1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error
trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi
medoutException: Commands 1710489661 to Host 3 timed out after 3600
2014-03-19 14:00:58,715 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null)
Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:00:58,715 WARN  [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902:
Timed out on null
2014-03-19 14:00:58,716 WARN  [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation
timed out: Commands 698
744902 to Host 4 timed out after 3600
2014-03-19 14:00:58,716 WARN  [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null)
Unable to obtain host 4 stati stics. 
2014-03-19 14:00:58,716 WARN  [cloud.server.StatsCollector] (StatsCollector-2:null) Received
invalid host stats for host
: 4
2014-03-19 14:01:22,978 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null)
Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:01:22,978 WARN  [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662:
Timed out on null
2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null)
Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands
1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error
trying to retrieve storage sta ts
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException:
Commands 1710489662 to Host 3 timed out after 3600
2014-03-19 14:01:35,900 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1
= [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has
been inactive for 3660
2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 =
[ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql]
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation
Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations
on VM[DomainRouter|r-4-VM]
2014-03-19 14:01:59,312 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:01:59,312 WARN  [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904:
Timed out on null
2014-03-19 14:01:59,312 WARN  [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation
timed out: Commands 698744904 to Host 4 timed out after 3600
2014-03-19 14:01:59,312 WARN  [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null)
Unable to obtain host 4 statistics. 
2014-03-19 14:01:59,312 WARN  [cloud.server.StatsCollector] (StatsCollector-1:null) Received
invalid host stats for host: 4
2014-03-19 14:02:07,765 WARN  [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The
task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,766 WARN  [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5)
Failed to deploy vm 4 with original planner, sending HAPlanner
2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable
to transition into Starting state due to Unable to transition to a new state from Starting
via StartRequested
2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining
why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM].  Retry=4
2014-03-19 14:02:07,812 WARN  [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The
task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720
2014-03-19 14:02:07,812 WARN  [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5)
Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM]
2014-03-19 14:02:07,812 WARN  [apache.cloudstack.alerts] (HA-Worker-0:work-5)  alertType::
9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM
which was running on host name: server2(id:1), availability zone: zone1,
 pod: pod1
2014-03-19 14:02:22,983 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:02:22,983 WARN  [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663:
Timed out on null
2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null)
Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands
1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error
trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException:
Commands 1710489663 to Host 3 timed out after 3600
2014-03-19 14:02:59,904 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:02:59,904 WARN  [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905:
Timed out on null
2014-03-19 14:02:59,905 WARN  [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation
timed out: Commands 698
744905 to Host 4 timed out after 3600
2014-03-19 14:02:59,905 WARN  [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null)
Unable to obtain host 4 statistics. 
2014-03-19 14:02:59,905 WARN  [cloud.server.StatsCollector] (StatsCollector-3:null) Received
invalid host stats for host: 4
2014-03-19 14:03:22,988 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:03:22,988 WARN  [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664:
Timed out on null
2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null)
Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands
1710489664 to Host 3 timed out after 3600
2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error
trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException:
Commands 1710489664 to Host 3 timed out after 3600
2014-03-19 14:04:00,500 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:04:00,500 WARN  [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906:
Timed out on null
2014-03-19 14:04:00,501 WARN  [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation
timed out: Commands 698744906 to Host 4 timed out after 3600
2014-03-19 14:04:00,501 WARN  [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null)
Unable to obtain host 4 statistics. 
2014-03-19 14:04:00,501 WARN  [cloud.server.StatsCollector] (StatsCollector-1:null) Received
invalid host stats for host: 4
2014-03-19 14:04:22,993 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:04:22,993 WARN  [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665:
Timed out on null
2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null)
Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands
1710489665 to Host 3 timed out after 3600
2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error
trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException:
Commands 1710489665 to Host 3 timed out after 3600
2014-03-19 14:05:01,092 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:05:01,092 WARN  [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907:
Timed out on null
2014-03-19 14:05:01,092 WARN  [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation
timed out: Commands 698744907 to Host 4 timed out after 3600
2014-03-19 14:05:01,092 WARN  [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null)
Unable to obtain host 4 statistics. 
2014-03-19 14:05:01,092 WARN  [cloud.server.StatsCollector] (StatsCollector-2:null) Received
invalid host stats for host
: 4
2014-03-19 14:05:23,000 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:05:23,000 WARN  [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666:
Timed out on null
2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null)
Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands
1710489666 to Host 3 timed out after 3600
2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error
trying to retrieve storage stats
com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException:
Commands 1710489666 to Host 3 timed out after 3600
2014-03-19 14:06:01,680 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:06:01,680 WARN  [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908:
Timed out on null
2014-03-19 14:06:01,680 WARN  [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation
timed out: Commands 698744908 to Host 4 timed out after 3600
2014-03-19 14:06:01,680 WARN  [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null)
Unable to obtain host 4 statistics. 
2014-03-19 14:06:01,681 WARN  [cloud.server.StatsCollector] (StatsCollector-1:null) Received
invalid host stats for host: 4
2014-03-19 14:06:23,004 INFO  [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null)
Could not find exception: com.cloud.exception.OperationTimedoutException in error code list
for exceptions
2014-03-19 14:06:23,004 WARN  [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667:
Timed out on null
2014-03-19 14:06:23,004 DEB





On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <Suresh.Sadhu@citrix.com> wrote:

Can you please  provide the logs and also  did  you notice  any exception in the management
log.


For deleting vm :
You can update the vm state in db  as Stopped and try to delete them from CS.

Regards
Sadhu




-----Original Message-----
From: Sugandh S [mailto:s.sugandh@rocketmail.com]
Sent: 19 March 2014 14:30
To: users@cloudstack.apache.org
Subject: vm stuck in starting state, unable to delete it

Hello all,

I am using CS 4.2 and my setup is as follows:

One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent.
Primary storage and secondary storage are also provided by this server via NFS. For primary
storage, export location is /export/primary and for secondary
 storage, it is /export/secondary.

Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent. 


Now, when I create vms they are stuck in starting state and I am unable to delete them.

Any and all help would be greatly appreciated.

Thanks ahead,
Sugandh
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message