cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Barnett <jbarn...@prokarma.com>
Subject RE: Unable to take Snapshot CS 4.2.1
Date Wed, 26 Nov 2014 15:19:52 GMT
The OS is CentOS 6.5, Storage is NFS not hosted on the controller.  I will also note that there
is two zones, with the same configuration just using different NFS devices one zone can take
snapshots but the one that can no longer take them has been working for 8 months with no issues.


Here is the section from the log file when I try to take a snapshot:


2014-11-26 07:00:48,308 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===START===  172.16.28.20
-- GET  command=createSnapshot&volumeid=ec6460c4-896c-46d1-8ae5-75d6431a6608&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014045703
2014-11-26 07:00:48,362 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-4:null) submit
async job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ], details: AsyncJobVO {id:147, userId:
2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 128, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo: {"id":"128","response":"json","sessionkey":"ysbTjq7whHzoePtvcRBRSzsJ6zw\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1417014045703","volumeid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","ctxAccountId":"2","ctxStartEventId":"5425"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode:
0, result: null, initMsid: 345048644703, completeMsid: null, lastUpdated: null, lastPolled:
null, created: null}
2014-11-26 07:00:48,363 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-65:job-147 =
[ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd
for job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]
2014-11-26 07:00:48,363 DEBUG [cloud.api.ApiServlet] (catalina-exec-4:null) ===END===  172.16.28.20
-- GET  command=createSnapshot&volumeid=ec6460c4-896c-46d1-8ae5-75d6431a6608&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014045703
2014-11-26 07:00:48,386 INFO  [user.snapshot.CreateSnapshotCmd] (Job-Executor-65:job-147 =
[ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) VOLSS: createSnapshotCmd starts:1417014048386
2014-11-26 07:00:48,437 DEBUG [agent.transport.Request] (Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace
]) Seq 7-1985284483: Sending  { Cmd , MgmtId: 345048644703, via: 7, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore: /Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"parentSnapshotPath":"2e7af1a5-a31c-4fe6-b2f7-21965c1a86b9","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore: /Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"wait":0}}]
}
2014-11-26 07:00:48,438 DEBUG [agent.transport.Request] (Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace
]) Seq 7-1985284483: Executing:  { Cmd , MgmtId: 345048644703, via: 7, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore: /Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"parentSnapshotPath":"2e7af1a5-a31c-4fe6-b2f7-21965c1a86b9","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore: /Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"wait":0}}]
}
2014-11-26 07:00:48,438 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-311:null) Seq
7-1985284483: Executing request
2014-11-26 07:00:48,438 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-311:null) Seq
7-1985284483: Response Received: 
2014-11-26 07:00:48,438 DEBUG [agent.transport.Request] (DirectAgent-311:null) Seq 7-1985284483:
Processing:  { Ans: , MgmtId: 345048644703, via: 7, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"b7c08601-0654-4749-b7e0-ccc380458e3d","id":0}},"result":true,"wait":0}}]
}
2014-11-26 07:00:48,438 DEBUG [agent.transport.Request] (Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace
]) Seq 7-1985284483: Received:  { Ans: , MgmtId: 345048644703, via: 7, Ver: v1, Flags: 10,
{ CreateObjectAnswer } }
2014-11-26 07:00:48,488 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-65:job-147
= [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) copyAsync inspecting src type SNAPSHOT copyAsync
inspecting dest type SNAPSHOT
2014-11-26 07:00:48,535 DEBUG [agent.transport.Request] (Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace
]) Seq 9-1742209940: Sending  { Cmd , MgmtId: 345048644703, via: 9, Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"b7c08601-0654-4749-b7e0-ccc380458e3d","volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore: /Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"parentSnapshotPath":"2e7af1a5-a31c-4fe6-b2f7-21965c1a86b9","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore: /Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/334","volume":{"uuid":"ec6460c4-896c-46d1-8ae5-75d6431a6608","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"055b61d2-5260-3856-92fc-48facf5dfbaf","id":2,"poolType":"VMFS","host":"VMFS
datastore: /Labs/CS-PRI-02","path":"/Labs/CS-PRI-02","port":0}},"name":"ROOT-297","size":21474836480,"path":"ROOT-297-000001","volumeId":334,"vmName":"i-2-297-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[CS-PRI-02]
i-2-297-VM/ROOT-297.vmdk\"]}","format":"OVA","id":334,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://172.16.25.84/data/CloudStackSEC2","_role":"Image"}},"vmName":"i-2-297-VM","name":"new-import_ROOT-297_20141126150048","hypervisorType":"VMware","id":128}},"executeInSequence":true,"wait":21600}}]
}
2014-11-26 07:00:51,404 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===START=== 
172.16.28.20 -- GET  command=queryAsyncJobResult&jobId=2007bb4e-03c9-4ff7-874d-e50641318ace&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014048864
2014-11-26 07:00:51,428 DEBUG [cloud.api.ApiServlet] (catalina-exec-25:null) ===END===  172.16.28.20
-- GET  command=queryAsyncJobResult&jobId=2007bb4e-03c9-4ff7-874d-e50641318ace&response=json&sessionkey=ysbTjq7whHzoePtvcRBRSzsJ6zw%3D&_=1417014048864
2014-11-26 07:00:51,840 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
SeqA 3-1302876: Processing Seq 3-1302876:  { Cmd , MgmtId: -1, via: 3, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n  \"connections\":
[]\n}","wait":0}}] }
2014-11-26 07:00:51,846 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-2:null)
SeqA 3-1302876: Sending Seq 3-1302876:  { Ans: , MgmtId: 345048644703, via: 3, Ver: v1, Flags:
100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2014-11-26 07:00:52,150 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq
9-1742209940: Processing:  { Ans: , MgmtId: 345048644703, via: 9, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"backup
snapshot exception: Exception: java.lang.Exception\nMessage: unable to prepare snapshot backup
directory\n","wait":0}}] }
2014-11-26 07:00:52,150 DEBUG [agent.transport.Request] (Job-Executor-65:job-147 = [ 2007bb4e-03c9-4ff7-874d-e50641318ace
]) Seq 9-1742209940: Received:  { Ans: , MgmtId: 345048644703, via: 9, Ver: v1, Flags: 110,
{ CopyCmdAnswer } }
2014-11-26 07:00:52,155 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-15:null)
Seq 9-1742209940: No more commands found
2014-11-26 07:00:52,171 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-65:job-147
= [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception: Exception: java.lang.Exception
Message: unable to prepare snapshot backup directory

	at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
	at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
	at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
	at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
	at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
	at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
	at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
	at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
	at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2014-11-26 07:00:52,184 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-65:job-147
= [ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Take snapshot: 334 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
	at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1034)
	at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
	at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
	at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
	at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
	at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception: Exception:
java.lang.Exception
Message: unable to prepare snapshot backup directory

	at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
	at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
	at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
	at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
	... 15 more
2014-11-26 07:00:52,184 DEBUG [user.snapshot.CreateSnapshotCmd] (Job-Executor-65:job-147 =
[ 2007bb4e-03c9-4ff7-874d-e50641318ace ]) Failed to create snapshot
org.apache.cloudstack.api.ServerApiException
	at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
	at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


-----Original Message-----
From: Andrija Panic [mailto:andrija.panic@gmail.com] 
Sent: Tuesday, November 25, 2014 11:55 PM
To: users@cloudstack.apache.org
Subject: Re: Unable to take Snapshot CS 4.2.1

Please attach the management logs for further invetigation...what OS, what storage, etc...

On 26 November 2014 at 01:51, Jeff Barnett <jbarnett@prokarma.com> wrote:

> When I try to take a snapshot now I just get an error like this, 
> Failed to create a snapshot due to an internal error creating snapshot for volume ...
> has anyone run into this, if so what was the fix, I tried restarting 
> the cloud services on the SSVM, but that did not help.
>
>
>
>


-- 

Andrija Panić
Mime
View raw message