cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Monos <tam...@veber.co.uk>
Subject Critical - DO NOT use 4.0 storage clean-up after upgrade from 3.0.2
Date Wed, 14 Nov 2012 15:31:25 GMT
Hi,

I had a successful update from 3.0.2->4.0.
Everything was perfect until I've enabled storage-clean-up in the global options.

At the next first time the storage clean-up thread run it marked all systemVM volumes/templates
on the primary storages for removal.
It has literally went through all my system VMs and first deleted their volumes than their
templates...
Then the system start to complain about file-not found messages, deploy failed...

I'm trying to recover from this but I cannot re-create the secondary storage VM due to:
2012-11-14 14:17:56,580 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There
is no secondary storage VM for secondary storage host nfs://192.168.1.2/share/watford_cloud

But I end up with:
2012-11-14 14:19:55,241 DEBUG [utils.script.Script] (catalina-exec-6:null) Looking for createvolume.sh
in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2012-11-14 14:19:55,241 INFO  [storage.template.DownloadManagerImpl] (catalina-exec-6:null)
createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2012-11-14 14:19:55,243 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null)
Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null)
Exit value is 1
2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null)
ln: creating symbolic link `/var/www/html/copy': No such file or directory
2012-11-14 14:19:55,304 WARN  [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null)
Error in linking  err=ln: creating symbolic link `/var/www/html/copy': No such file or directory
2012-11-14 14:19:55,305 INFO  [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null)
Unable to fully initialize the agent because no StartupCommands are returned
2012-11-14 14:19:55,305 INFO  [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null)
server resources successfully discovered by SecondaryStorage
2012-11-14 14:19:55,305 WARN  [cloud.api.ApiDispatcher] (catalina-exec-6:null) class com.cloud.api.ServerApiException
: Failed to add secondary storage

This is issue is likely caused by the update procedure due to the systemVM were not using
the latest template.
They were not re-deployed just re-started with the new systemVM template and possible some
references were not updated in other DB tables so the clean-up script found them as not used
and destroyed them all.
I'm lucky enough I always have test environment for everything.

Regards

Tamas Monos                                               DDI         +44(0)2034687012
Chief Technical                                             Office    +44(0)2034687000
Veber: The Hosting Specialists               Fax         +44(0)871 522 7057
http://www.veber.co.uk<http://www.veber.co.uk/>

Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message