Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7F6E710A96 for ; Mon, 11 Nov 2013 16:37:18 +0000 (UTC) Received: (qmail 51448 invoked by uid 500); 11 Nov 2013 16:37:18 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 51273 invoked by uid 500); 11 Nov 2013 16:37:18 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 51262 invoked by uid 500); 11 Nov 2013 16:37:17 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 51256 invoked by uid 99); 11 Nov 2013 16:37:17 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Nov 2013 16:37:17 +0000 Date: Mon, 11 Nov 2013 16:37:17 +0000 (UTC) From: "Milamber (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] Milamber updated CLOUDSTACK-4892: --------------------------------- Priority: Critical (was: Major) > KVM snapshots are failing on CLVM > --------------------------------- > > Key: CLOUDSTACK-4892 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-489= 2 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: KVM, Snapshot > Affects Versions: 4.2.0 > Environment: CentOS 6.4, KVM, CLVM > Reporter: Ivan Kozlov > Priority: Critical > > Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes cre= ating snaphot is successful. > Snapshot logical volume is created and not deleted. > When running snaphot with only single host snapshot is created normaly. G= uess snapshot backup is trying access snapshot LV from host on which snapsh= ot LV is not opened. > Here is management log: > 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina= -exec-10:null) submit async job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e3= 7c ], details: AsyncJobVO {id:41, userId: 2, accountId: 2, sessionKey: null= , instanceType: Snapshot, instanceId: 10, cmd: org.apache.cloudstack.api.co= mmand.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":= "10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cm= dEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"13821= 06777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":= "2","ctxStartEventId":"126"}, cmdVersion: 0, callbackType: 0, callbackAddre= ss: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsi= d: 161342718518, completeMsid: null, lastUpdated: null, lastPolled: null, c= reated: null} > 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Exec= utor-22:job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing org.= apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41 = =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ] > 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd] (Job-Exec= utor-22:job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS: createS= napshotCmd starts:1382106778549 > 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request] (Job-Executor-22:= job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 1-111542657: Sendi= ng { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags: 100011, [{"org.ap= ache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.c= loudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43c= c-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstac= k.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281c= b04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port= ":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c2= 36a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","i= d":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc= -d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataSto= re":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8= c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost",= "path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5= _20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}] } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] (AgentManager-Han= dler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId: 161342718518, v= ia: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateO= bjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{= "path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-= 4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}] } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] (Job-Executor-22:= job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 1-111542657: Recei= ved: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1, Flags: 10, { CreateOb= jectAnswer } } > 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy] = (Job-Executor-22:job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyA= sync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT > 2013-10-18 17:33:00,547 DEBUG [agent.transport.Request] (Job-Executor-22:= job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 4-1918238786: Send= ing { Cmd , MgmtId: 161342718518, via: 4, Ver: v1, Flags: 100111, [{"org.a= pache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudst= ack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e5= 5-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","volume":{"uuid":= "560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"or= g.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4= d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"= /vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-= d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2= ,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/v= g_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1aba= df2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO= ":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","= host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name= ":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"destTO= ":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2= /5","volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"R= OOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"u= uid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host"= :"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934= 592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-= 2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parent= SnapshotPath":"snapshots/2/5/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataSto= re":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/sec= ondary","_role":"Image"}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131= 018143258","hypervisorType":"KVM","id":10}},"executeInSequence":true,"wait"= :21600}}] } > 2013-10-18 17:33:01,069 DEBUG [agent.transport.Request] (AgentManager-Han= dler-8:null) Seq 4-1918238786: Processing: { Ans: , MgmtId: 161342718518, = via: 4, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyC= mdAnswer":{"result":false,"details":"Disk /dev/vg_primary/4f3e8cfc-d3be-4e5= 5-bc13-5c236a689c83 has no snapshot called 5ece5182b7f6f10ef1b66b558f3d0bbb= .","wait":0}}] } > 2013-10-18 17:33:01,070 DEBUG [agent.manager.AgentAttache] (AgentManager-= Handler-8:null) Seq 4-1918238786: No more commands found > 2013-10-18 17:33:01,070 DEBUG [agent.transport.Request] (Job-Executor-22:= job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 4-1918238786: Rece= ived: { Ans: , MgmtId: 161342718518, via: 4, Ver: v1, Flags: 110, { CopyCm= dAnswer } } > 2013-10-18 17:33:01,336 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job= -Executor-22:job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Failed to= create snapshot > com.cloud.utils.exception.CloudRuntimeException: Disk /dev/vg_primary/4f3= e8cfc-d3be-4e55-bc13-5c236a689c83 has no snapshot called 5ece5182b7f6f10ef1= b66b558f3d0bbb. > at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.bac= kupSnapshot(SnapshotServiceImpl.java:280) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(Sn= apshotManagerImpl.java:1013) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$= InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:12= 5) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSna= pshot(VolumeServiceImpl.java:1307) > at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManager= Impl.java:2720) > at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshot= Cmd.execute(CreateSnapshotCmd.java:170) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.= java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1146) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > 2013-10-18 17:33:01,427 DEBUG [storage.volume.VolumeServiceImpl] (Job-Exe= cutor-22:job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Take snapshot= : 5 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapsho= t > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(Sn= apshotManagerImpl.java:1040) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$= InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:12= 5) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSna= pshot(VolumeServiceImpl.java:1307) > at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManager= Impl.java:2720) > at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshot= Cmd.execute(CreateSnapshotCmd.java:170) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.= java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1146) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > Caused by: com.cloud.utils.exception.CloudRuntimeException: Disk /dev/vg_= primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83 has no snapshot called 5ece518= 2b7f6f10ef1b66b558f3d0bbb. > at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.bac= kupSnapshot(SnapshotServiceImpl.java:280) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(Sn= apshotManagerImpl.java:1013) > ... 16 more > 2013-10-18 17:33:01,509 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Exec= utor-22:job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Complete async= job-41 =3D [ 88ec27d7-78af-4664-a01b-eeca4469e37c ], jobStatus: 2, resultC= ode: 530, result: Error Code: 530 Error text: Failed to create snapshot due= to an internal error creating snapshot for volume 5 -- This message was sent by Atlassian JIRA (v6.1#6144)