incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Monos <tam...@veber.co.uk>
Subject RE: Permanent snapshot problem bound to a specific volume
Date Thu, 14 Jun 2012 17:06:12 GMT
Hi,

Any suggestions how to attach a debugger to a running CS management server to find out what
is causing the java.lang.ArrayIndexOutOfBoundsException?

Regards

Tamas Monos                                               DDI         +44(0)2034687012
Chief Technical                                             Office    +44(0)2034687000
Veber: The Hosting Specialists               Fax         +44(0)871 522 7057
http://www.veber.co.uk

Follow us on Twitter: www.twitter.com/veberhost
Follow us on Facebook: www.facebook.com/veberhost


-----Original Message-----
From: Tamas Monos [mailto:tamasm@veber.co.uk] 
Sent: 14 June 2012 12:58
To: cloudstack-dev@incubator.apache.org
Subject: Permanent snapshot problem bound to a specific volume

Hi,

Last night I tried to create a snapshot manually from a volume.
The mangement server was performing the operation unfortunately got power cycled through the
process :( This caused preventing me to do any further snapshots as the system said backup
is already in progress on that volume. Please try again later.
After hours it was still stuck saying the same (while vmware was not doing anything at all)
so I went to the 'snapshots' table in the DB and marked its state from 'BackingUP' to 'Error'.
This allowed me to create snapshots again and started to export the OVF template just fine
(seen the progress via vcenter). After about an hour the export failed around 15%.
Since then I cannot create any snapshots of that VM while can make from others with no problems.

The log:

2012-06-14 10:42:19,918 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-8:job-1063)
Cleanup failed due to BackupSnapshotCommand exception: java.lang.Exception: Unable to finish
the whole process to package as a OVA file
2012-06-14 10:42:19,918 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063)
The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception:
java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:42:19,919 WARN  [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-3:null)
Cleanup failed due to BackupSnapshotCommand exception: java.lang.Exception: Unable to finish
the whole process to package as a OVA file

2012-06-14 10:44:39,289 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq
10-577440210: Processing:  { Ans: , MgmtId: 345049202747, via: 10, Ver: v1, Flags: 10, [{"BackupSnapshotAnswer":{"full":true,"result":false,"details":"BackupSnapshotCommand
exception: java.lang.ArrayIndexOutOfBoundsException: 0\ncom.cloud.hypervisor.vmware.util.VmwareHelper.prepareDiskDevice(VmwareHelper.java:220)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromDiskChain(VirtualMachineMO.java:1474)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromCurrentSnapshot(VirtualMachineMO.java:1450)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.exportVolumeToSecondaryStroage(VmwareStorageManagerImpl.java:747)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.backupSnapshotToSecondaryStorage(VmwareStorageManagerImpl.java:714)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:217)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.execute(VmwareSecondaryStorageResourceHandler.java:109)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.executeRequest(VmwareSecondaryStorageResourceHandler.java:67)\ncom.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:50)\ncom.cloud.agent.Agent.processRequest(Agent.java:517)\ncom.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)\ncom.cloud.utils.nio.Task.run(Task.java:79)\njava.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\njava.lang.Thread.run(Thread.java:662)\n","wait":0}}]
}

2012-06-14 10:44:39,289 DEBUG [agent.transport.Request] (Job-Executor-8:job-1063) Seq 10-577440210:
Received:  { Ans: , MgmtId: 345049202747, via: 10, Ver: v1, Flags: 10, { BackupSnapshotAnswer
} }
2012-06-14 10:44:39,323 WARN  [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-8:job-1063)
Cleanup failed due to BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException:
0
2012-06-14 10:44:39,324 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063)
The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception:
java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:44:39,324 WARN  [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-2:null)
Cleanup failed due to BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException:
0
2012-06-14 10:46:39,325 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063)
Retrying...
2012-06-14 10:46:39,329 WARN  [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063)
Storage unavailable
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:201] is unreachable:
Unable to send command to the pool 201 due to there is no enabled hosts up in this cluster
2012-06-14 10:46:39,330 WARN  [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063)
Failed to back up snapshot on secondary storage, deleting the record from the DB
2012-06-14 10:46:39,377 ERROR [cloud.api.ApiDispatcher] (Job-Executor-8:job-1063) Exception
while executing CreateSnapshotCmd:
com.cloud.utils.exception.CloudRuntimeException: Created snapshot: com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$aaba1dbd@44374434<mailto:com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$aaba1dbd@44374434>
on primary but failed to backup on secondary

At the end it says Storage unavailable... I can parallel create other snapshots with no problem
so the storage is there.
That java.lang.ArrayIndexOutOfBoundsException is quite scary though.

Any advice or magic could sort this problem would be appreciated.

Regards

Tamas Monos                                               DDI         +44(0)2034687012
Chief Technical                                             Office    +44(0)2034687000
Veber: The Hosting Specialists               Fax         +44(0)871 522 7057
http://www.veber.co.uk<http://www.veber.co.uk/>

Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>



Mime
View raw message