cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: How to recover entire self contained CloudStack after major network changes?
Date Tue, 29 Oct 2013 15:59:40 GMT
my bad for secondary storage -  there is no maintance mode for that :)

I was just refering to line:

host=10.97.38.10 - which is hopefully new IP address :)

Since you got Server capactity issues, I suggest folowing, as temp solution:

Check CPU alocation, RAM alocation, and increase overprovisioning factor at
POD level (at least this is in 4.2, or only in global config as was in
4.0.0), so if you are running out of CPU/RAM resopurces, you might be able
to fix this this way.

I had this exact type of problem (SSVM not starting), when upgraded from
4,0 to 4.2,because of bad SSVM template  - I had to download
http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2(previously
I had very old, which came with Cs 4.0.0) - but if you did not
upgrade your CS, then I suppose this is not the case...

Best





On 29 October 2013 16:42, Adam <adam.scarcella@gmail.com> wrote:

> Hi Andrija,
>
> Thanks for your reply.
>
> Here is the output of my agent.properties file (which looks correct to me):
>
> [root@cs-east-prod1 ~]# cat /etc/cloudstack/agent/agent.properties
> #Storage
> #Tue Oct 29 10:15:49 GMT-05:00 2013
> guest.network.device=cloudbr0
> workers=5
> private.network.device=cloudbr0
> port=8250
> resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
> pod=1
> zone=1
> guid=db24e08f-ac6e-3974-b223-274d49af52d0
> cluster=1
> public.network.device=cloudbr0
> local.storage.uuid=296f0404-9658-4269-bc24-64b1247d5e3f
> domr.scripts.dir=scripts/network/domr/kvm
> host=10.97.38.10
> LibvirtComputingResource.id=10
>
> [root@cs-east-prod1 ~]# ifconfig cloudbr0
> cloudbr0  Link encap:Ethernet  HWaddr AC:16:2D:84:D4:24
>           inet addr:10.97.38.10  Bcast:10.97.38.255  Mask:255.255.255.0
>           inet6 addr: fe80::ae16:2dff:fe84:d424/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:10216 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:10343 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:1605339 (1.5 MiB)  TX bytes:24242332 (23.1 MiB)
>
>
> I have 4.5 TB free on my 5.7 TB RAID so I haven't even come close to the
> "pool.storage.allocated.capacity.disablethreshold" or
> "pool.storage.capacity.disablethreshold" of "0.85"
>
> [root@cs-east-prod1 ~]# df -kh
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/vg_cloudstackuse-lv_root
>                       5.7T  921G  4.5T  17% /
> tmpfs                  64G   72K   64G   1% /dev/shm
> /dev/sda1             485M  118M  342M  26% /boot
> 10.97.38.10:/export/primary
>                       5.7T  921G  4.5T  17%
> /mnt/910365e6-a298-37b8-904f-2b49198c8286
> [root@cs-east-prod1 ~]# cd /export/
> [root@cs-east-prod1 export]# ls -l
> total 8
> drwxrwxrwx. 3 root root 4096 Oct 27 19:18 primary
> drwxrwxrwx. 5 root root 4096 Jul 16 11:27 secondary
> [root@cs-east-prod1 export]# du -sh *
> 238G    primary
> 110G    secondary
> [root@cs-east-prod1 export]#
>
> I've manually destroyed the SSVM several times and yes CS does attempt to
> create a new one, but the same issue occurs. Regardless of whether I
> destroy the SSVM or if CS does it, it just keeps looping.
>
> I'm not exactly sure how to check if my secondary storage is in Maintenance
> Mode. I know I can enable Maintenance Mode of my Primary Storage, but how
> to check for the Secondary Storage?
>
> Best Regards,
>
>
>
> Adam Scarcella
>
>
> On Tue, Oct 29, 2013 at 11:16 AM, Andrija Panic <andrija.panic@gmail.com
> >wrote:
>
> > I suppose, you can recheck the network, and also check the AGENT
> > configuration file, make changes
> > there: /etc/cloudstack/agent/agent.properties
> >
> > From logs:
> >
> > InsufficientServerCapacityException: Unable to create
> > > a deployment for VM.....
> >
> > I had this kind of error, when my storage pools were disabled once, and
> > also when I have reached "disable capacity treshold" - my storage pool
> was
> > over 85% full.
> >
> > So check if you secondaru storge is in maintance mode, or similar...
> >
> > Also, destroying SSVM will force CS into making new one...
> >
> >
> > On 29 October 2013 16:09, Adam <adam.scarcella@gmail.com> wrote:
> >
> > > Hi Geoff,
> > >
> > > Thanks for the reply, I was able to manually mount the secondary
> storage
> > > NFS:
> > >
> > > [root@cs-east-prod1 secondaryStorage]# hostname
> > > cs-east-prod1
> > > [root@cs-east-prod1 secondaryStorage]# hostname --fqdn
> > > cs-east-prod1.na.tibco.com
> > > [root@cs-east-prod1 secondaryStorage]#
> > > [root@cs-east-prod1 ~]# mkdir /secondaryStorage
> > > [root@cs-east-prod1 ~]# mount -t nfs cs-east-prod1:/export/secondary
> > > /secondaryStorage
> > > [root@cs-east-prod1 ~]# cd /
> > > [root@cs-east-prod1 /]# ls -l
> > > total 120
> > > dr-xr-xr-x.   2 root root  4096 Oct 23 04:48 bin
> > > dr-xr-xr-x.   5 root root  3072 Oct 22 21:10 boot
> > > drwxr-xr-x.  10 root root  4096 Jun 25 10:23 cgroup
> > > drwxr-xr-x.  19 root root  3960 Oct 29 10:38 dev
> > > drwxr-xr-x. 121 root root 12288 Oct 29 11:07 etc
> > > drwxrwxrwx.   4 root root  4096 Jul  3 20:10 export
> > > drwxr-xr-x.   3 root root  4096 Jul  3 19:14 home
> > > lrwxrwxrwx.   1 root root    25 Aug 28 15:46 Installers ->
> > > /var/www/html/Installers/
> > > lrwxrwxrwx.   1 root root    19 Jul  5 20:58 ISOs ->
> /var/www/html/ISOs/
> > > dr-xr-xr-x.  11 root root  4096 Jul  3 19:02 lib
> > > dr-xr-xr-x.  10 root root 12288 Oct 28 03:33 lib64
> > > drwx------.   2 root root 16384 Jul  3 18:44 lost+found
> > > drwxr-xr-x.   3 root root  4096 Oct 27 12:05 media
> > > drwxr-xr-x.   2 root root     0 Oct 29 10:08 misc
> > > drwxr-xr-x.  39 root root  4096 Sep  6 12:07 mnt
> > > drwxr-xr-x.   2 root root     0 Oct 29 10:08 net
> > > drwxr-xr-x.   3 root root  4096 Jul  3 19:05 opt
> > > drwxr-xr-x.   2 root root  4096 Oct 27 10:06 primarymount
> > > dr-xr-xr-x. 557 root root     0 Oct 29 10:08 proc
> > > dr-xr-x---.  35 root root  4096 Oct 29 10:09 root
> > > dr-xr-xr-x.   2 root root 12288 Oct 28 03:33 sbin
> > > drwxr-xr-x.   2 root root  4096 Oct 27 10:07 secondarymount
> > > drwxrwxrwx.   5 root root  4096 Jul 16 11:27 secondaryStorage
> > > drwxr-xr-x.   7 root root     0 Oct 29 10:08 selinux
> > > drwxr-xr-x.   2 root root  4096 Sep 23  2011 srv
> > > drwxr-xr-x.  13 root root     0 Oct 29 10:08 sys
> > > drwxrwxrwt.  16 root root  4096 Oct 29 11:07 tmp
> > > drwxr-xr-x.  13 root root  4096 Jul  3 18:55 usr
> > > drwxr-xr-x.  23 root root  4096 Jul  3 19:45 var
> > > [root@cs-east-prod1 /]# cd secondaryStorage/
> > > [root@cs-east-prod1 secondaryStorage]# ls -l
> > > total 12
> > > drwxrwxrwx. 2 root root 4096 Jul 16 11:27 snapshots
> > > drwxrwxrwx. 3 root root 4096 Jul 16 11:10 template
> > > drwxrwxrwx. 3 root root 4096 Jul 22 09:38 volumes
> > > [root@cs-east-prod1 secondaryStorage]#
> > >
> > >
> > >
> > > Best Regards,
> > >
> > >
> > >
> > > Adam Scarcella
> > >
> > >
> > > On Tue, Oct 29, 2013 at 10:54 AM, Geoff Higginbottom <
> > > geoff.higginbottom@shapeblue.com> wrote:
> > >
> > > > Adam,
> > > >
> > > > System VMs looping and failing to create is often caused by the Host
> > > being
> > > > unable to access Sec Storage.
> > > >
> > > > Do a manual mount of the Sec Storage NFS on the Host to confirm it
> can
> > > > access it.
> > > >
> > > > Regards
> > > >
> > > > Geoff Higginbottom
> > > >
> > > > D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
> > > >
> > > > geoff.higginbottom@shapeblue.com
> > > >
> > > > -----Original Message-----
> > > > From: Adam [mailto:adam.scarcella@gmail.com]
> > > > Sent: 29 October 2013 14:44
> > > > To: users@cloudstack.apache.org
> > > > Subject: Re: How to recover entire self contained CloudStack after
> > major
> > > > network changes?
> > > >
> > > > I exported the db as a *.sql file, did search/replace and that seemed
> > to
> > > > help a lot, but still not 100% out of the woods. Now I'm hitting the
> > > issue
> > > > where the System VMs (SecondaryStorageVm & ConsoleProxy) do not
> start.
> > > They
> > > > just keep looping from creation to being expunged.
> > > >
> > > > My Zone is enabled, Pod is enabled, Cluster is enabled, Host is Up,
> > > > Primary Storage is online:
> > > >
> > > > [root@cs-east-prod1 ~]# df -kh
> > > > Filesystem            Size  Used Avail Use% Mounted on
> > > > /dev/mapper/vg_cloudstackuse-lv_root
> > > >                       5.7T  921G  4.5T  17% /
> > > > tmpfs                  64G   72K   64G   1% /dev/shm
> > > > /dev/sda1             485M  118M  342M  26% /boot
> > > > 10.97.38.10:/export/primary
> > > >                       5.7T  921G  4.5T  17%
> > > > /mnt/910365e6-a298-37b8-904f-2b49198c8286
> > > >
> > > > Here is an excerpt from my most recent log:
> > > >
> > > > 2013-10-29 09:22:23,389 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > > 2013-10-29 09:22:23,391 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > > 2013-10-29 09:22:23,441 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 routers to update status.
> > > > 2013-10-29 09:22:23,443 DEBUG
> > > > [network.router.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> > > > 2013-10-29 09:22:26,872 DEBUG [agent.manager.AgentManagerImpl]
> > > > (AgentManager-Handler-5:null) Ping from 10
> > > > 2013-10-29 09:22:38,509 DEBUG [cloud.server.StatsCollector]
> > > > (StatsCollector-2:null) VmStatsCollector is running...
> > > > 2013-10-29 09:22:38,708 DEBUG [cloud.server.StatsCollector]
> > > > (StatsCollector-3:null) StorageCollector is running...
> > > > 2013-10-29 09:22:38,716 INFO
> >  [storage.endpoint.DefaultEndPointSelector]
> > > > (StatsCollector-3:null) No running ssvm is found, so command will be
> > sent
> > > > to LocalHostEndPoint
> > > > 2013-10-29 09:22:44,969 DEBUG [cloud.server.StatsCollector]
> > > > (StatsCollector-1:null) HostStatsCollector is running...
> > > > 2013-10-29 09:22:45,587 DEBUG [agent.transport.Request]
> > > > (StatsCollector-1:null) Seq 10-768344083: Received:  { Ans: , MgmtId:
> > > > 86691596708691, via: 10, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > > > 2013-10-29 09:22:51,802 DEBUG
> > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > Zone
> > > 1
> > > > is ready to launch secondary storage VM
> > > > 2013-10-29 09:22:51,807 INFO
> > > >  [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > > (secstorage-1:null)
> > > > No running secondary storage vms found in datacenter id=1, starting
> one
> > > > 2013-10-29 09:22:51,814 INFO
> > > >  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> No
> > > > stopped secondary storage vm is available, need to allocate a new
> > > secondary
> > > > storage vm
> > > > 2013-10-29 09:22:51,817 DEBUG
> > > > [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > Assign
> > > > secondary storage vm from a newly started instance for request from
> > data
> > > > center : 1
> > > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering
> [6-Guest-DefaultSharedNetworkOfferingWithSGService]:
> > > > Ntwk[204|Guest|6]
> > > > 2013-10-29 09:22:51,837 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering [3-Control-System-Control-Network]:
> > Ntwk[202|Control|3]
> > > > 2013-10-29 09:22:51,842 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,846 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering [2-Management-System-Management-Network]:
> > > > Ntwk[201|Management|2]
> > > > 2013-10-29 09:22:51,847 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Found existing network configuration for offering
> > > > [Network Offering [4-Storage-System-Storage-Network]:
> > Ntwk[203|Storage|4]
> > > > 2013-10-29 09:22:51,851 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Releasing lock for
> > > > Acct[6f5c60f0-ee29-11e2-b62c-ac162d84d424-system]
> > > > 2013-10-29 09:22:51,860 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocating entries for VM:
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,865 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocating nics for
> > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,866 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[204|Guest|6] with requested profile
> > > > NicProfile[0-0-null-null-null
> > > > 2013-10-29 09:22:51,881 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[202|Control|3] with requested profile null
> > > > 2013-10-29 09:22:51,890 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[201|Management|2] with requested profile null
> > > > 2013-10-29 09:22:51,903 DEBUG [cloud.network.NetworkManagerImpl]
> > > > (secstorage-1:null) Allocating nic for vm
> > > VM[SecondaryStorageVm|s-4930-VM]
> > > > in network Ntwk[203|Storage|4] with requested profile null
> > > > 2013-10-29 09:22:51,915 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocating disks for
> > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,922 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Allocation completed for VM:
> > > > VM[SecondaryStorageVm|s-4930-VM]
> > > > 2013-10-29 09:22:51,925 DEBUG
> > > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > > (secstorage-1:null) received secondary storage vm alert
> > > > 2013-10-29 09:22:51,925 DEBUG
> > > [cloud.alert.SecondaryStorageVmAlertAdapter]
> > > > (secstorage-1:null) New secondary storage vm created, zone:
> > > EastProdZone1,
> > > > secStorageVm: s-4930-VM, public IP: null, private IP: null
> > > > 2013-10-29 09:22:51,942 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (secstorage-1:null) VM state transitted from :Stopped to Starting
> with
> > > > event: StartRequestedvm's original host id: null new host id: null
> host
> > > id
> > > > before state transition: null
> > > > 2013-10-29 09:22:51,942 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Successfully transitioned to start state for
> > > > VM[SecondaryStorageVm|s-4930-VM] reservation id =
> > > > 3b2435a0-9e32-4777-a42f-a766a375bc1b
> > > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId:
> null
> > > > 2013-10-29 09:22:51,944 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts:
> > null
> > > > 2013-10-29 09:22:51,945 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts:
> > null
> > > > 2013-10-29 09:22:51,946 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) DeploymentPlanner allocation algorithm:
> > > >
> com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > > 2013-10-29 09:22:51,946 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Trying to allocate a host and storage pools from
> > > dc:1,
> > > > pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
> > > > 2013-10-29 09:22:51,946 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Is ROOT volume READY (pool already allocated)?:
> No
> > > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Searching all possible resources under this
> Zone: 1
> > > > 2013-10-29 09:22:51,946 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Listing clusters in order of aggregate capacity,
> > that
> > > > have (atleast one host with) enough CPU and RAM capacity under this
> > > Zone: 1
> > > > 2013-10-29 09:22:51,958 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > > 2013-10-29 09:22:51,958 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> >  pod:1
> > > >  cluster:1
> > > > 2013-10-29 09:22:51,962 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) FirstFitAllocator has 0 hosts
> > to
> > > > check for allocation: []
> > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Found 0 hosts for allocation
> > > after
> > > > prioritization: []
> > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > Ram=256
> > > > 2013-10-29 09:22:51,964 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (secstorage-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > > > suitable hosts
> > > > 2013-10-29 09:22:51,965 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) No suitable hosts found
> > > > 2013-10-29 09:22:51,965 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) No suitable hosts found under this Cluster: 1
> > > > 2013-10-29 09:22:51,968 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (secstorage-1:null) Could not find suitable Deployment Destination
> for
> > > > this VM under any clusters, returning.
> > > > 2013-10-29 09:22:51,968 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Searching all possible resources under this
> Zone: 1
> > > > 2013-10-29 09:22:51,969 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Listing clusters in order of aggregate capacity,
> > that
> > > > have (atleast one host with) enough CPU and RAM capacity under this
> > > Zone: 1
> > > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) Removing from the clusterId list these clusters
> > from
> > > > avoid set: [1]
> > > > 2013-10-29 09:22:51,973 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (secstorage-1:null) No clusters found after removing disabled
> clusters
> > > and
> > > > clusters in avoid list, returning.
> > > > 2013-10-29 09:22:51,981 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (secstorage-1:null) VM state transitted from :Starting to Stopped
> with
> > > > event: OperationFailedvm's original host id: null new host id: null
> > host
> > > > id before state transition: null
> > > > 2013-10-29 09:22:51,985 WARN
> > > >  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > > Exception while trying to start secondary storage vm
> > > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > create
> > > > a deployment for VM[SecondaryStorageVm|s-4930-VM]Scope=interface
> > > > com.cloud.dc.DataCenter; id=1
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> > > >         at
> > > >
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> > > >         at
> > > >
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > >         at
> > > >
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > >         at
> > > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > >         at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > >         at
> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >         at java.lang.Thread.run(Thread.java:744)
> > > > 2013-10-29 09:22:51,987 INFO
> > > >  [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null)
> > > > Unable to start secondary storage vm for standby capacity,
> secStorageVm
> > > vm
> > > > Id :
> > > > 4930, will recycle it and start a new one
> > > > 2013-10-29 09:22:51,987 INFO
> > > >  [cloud.secstorage.PremiumSecondaryStorageManagerImpl]
> > > (secstorage-1:null)
> > > > Primary secondary storage is not even started, wait until next turn
> > > > 2013-10-29 09:22:52,009 DEBUG
> > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Zone 1 is ready to launch console proxy
> > > > 2013-10-29 09:22:52,009 DEBUG
> > > [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Expand console proxy standby capacity for zone
> > > > EastProdZone1
> > > > 2013-10-29 09:22:52,012 INFO
> > >  [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Found a stopped console proxy, bring it up to
> > > > running pool. proxy vm id : 4271
> > > > 2013-10-29 09:22:52,020 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (consoleproxy-1:null) VM state transitted from :Stopped to Starting
> > with
> > > > event: StartRequestedvm's original host id: null new host id: null
> host
> > > id
> > > > before state transition: null
> > > > 2013-10-29 09:22:52,020 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:null) Successfully transitioned to start state for
> > > > VM[ConsoleProxy|v-4271-VM] reservation id =
> > > > 11861d83-fc09-4b25-8c07-bcb6ad8e7f54
> > > > 2013-10-29 09:22:52,021 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId:
> > null
> > > > 2013-10-29 09:22:52,022 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null,
> hosts:
> > > null
> > > > 2013-10-29 09:22:52,023 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Deploy avoids pods: null, clusters: null,
> hosts:
> > > null
> > > > 2013-10-29 09:22:52,024 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) DeploymentPlanner allocation algorithm:
> > > >
> com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_d4b92e44@713baeaa
> > > > 2013-10-29 09:22:52,024 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Trying to allocate a host and storage pools
> from
> > > > dc:1, pod:null,cluster:null, requested cpu: 500, requested ram:
> > > 1073741824
> > > > 2013-10-29 09:22:52,024 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?:
> > No
> > > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Searching all possible resources under this
> > Zone: 1
> > > > 2013-10-29 09:22:52,024 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Listing clusters in order of aggregate
> capacity,
> > > > that have (atleast one host with) enough CPU and RAM capacity under
> > this
> > > > Zone: 1
> > > > 2013-10-29 09:22:52,031 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Checking resources in Cluster: 1 under Pod: 1
> > > > 2013-10-29 09:22:52,031 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for hosts in dc: 1
> > >  pod:1
> > > >  cluster:1
> > > > 2013-10-29 09:22:52,033 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) FirstFitAllocator has 0
> hosts
> > > to
> > > > check for allocation: []
> > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Found 0 hosts for
> allocation
> > > > after prioritization: []
> > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Looking for speed=500Mhz,
> > > > Ram=1024
> > > > 2013-10-29 09:22:52,034 DEBUG [allocator.impl.FirstFitAllocator]
> > > > (consoleproxy-1:FirstFitRoutingAllocator) Host Allocator returning 0
> > > > suitable hosts
> > > > 2013-10-29 09:22:52,034 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) No suitable hosts found
> > > > 2013-10-29 09:22:52,034 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) No suitable hosts found under this Cluster: 1
> > > > 2013-10-29 09:22:52,036 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl]
> > > > (consoleproxy-1:null) Could not find suitable Deployment Destination
> > for
> > > > this VM under any clusters, returning.
> > > > 2013-10-29 09:22:52,036 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Searching all possible resources under this
> > Zone: 1
> > > > 2013-10-29 09:22:52,037 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Listing clusters in order of aggregate
> capacity,
> > > > that have (atleast one host with) enough CPU and RAM capacity under
> > this
> > > > Zone: 1
> > > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) Removing from the clusterId list these clusters
> > > from
> > > > avoid set: [1]
> > > > 2013-10-29 09:22:52,039 DEBUG [cloud.deploy.FirstFitPlanner]
> > > > (consoleproxy-1:null) No clusters found after removing disabled
> > clusters
> > > > and clusters in avoid list, returning.
> > > > 2013-10-29 09:22:52,043 DEBUG [cloud.capacity.CapacityManagerImpl]
> > > > (consoleproxy-1:null) VM state transitted from :Starting to Stopped
> > with
> > > > event: OperationFailedvm's original host id: null new host id: null
> > host
> > > > id before state transition: null
> > > > 2013-10-29 09:22:52,045 WARN
> > >  [cloud.consoleproxy.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:null) Exception while trying to start console proxy
> > > > com.cloud.exception.InsufficientServerCapacityException: Unable to
> > create
> > > > a deployment for VM[ConsoleProxy|v-4271-VM]Scope=interface
> > > > com.cloud.dc.DataCenter; id=1
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
> > > >         at
> > > >
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
> > > >         at
> > > >
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
> > > >         at
> > > >
> > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> > > >         at
> > > >
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> > > >         at
> > > > com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> > > >         at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > >         at
> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >         at java.lang.Thread.run(Thread.java:744)
> > > >
> > > >
> > > > What am I missing? How can I get these two system VMs online?
> > > >
> > > > Best Regards,
> > > >
> > > >
> > > >
> > > > Adam Scarcella
> > > >
> > > >
> > > > On Mon, Oct 28, 2013 at 8:07 PM, Adam <adam.scarcella@gmail.com>
> > wrote:
> > > >
> > > > > It's worth a shot. I've gone through the db manually, and changed
> all
> > > > > the places where I could see the old IPs, but your way is much more
> > > > efficient.
> > > > > Doing it now and will report back soon. Thanks.
> > > > >
> > > > > Best Regards,
> > > > >
> > > > >
> > > > >
> > > > > Adam Scarcella
> > > > >
> > > > >
> > > > > On Mon, Oct 28, 2013 at 6:53 PM, Andrija Panic <
> > > andrija.panic@gmail.com
> > > > >wrote:
> > > > >
> > > > >> one of the crazy tries:
> > > > >> backup your database, search for all IP addresses inside SQL
file,
> > > > >> map old to new IP addresses (old1 --> new1, old2 -->new2...)
and
> > then
> > > > >> do search/replace each old one with new one.
> > > > >> This is for netowrking/ip addresses (guest, public,etc), same
for
> > NFS
> > > > >> serves, same for NFS host, etc...
> > > > >>
> > > > >> Not sure if that is going to make you UP again, but...
> > > > >>
> > > > >>
> > > > >> On 28 October 2013 22:59, Adam <adam.scarcella@gmail.com>
wrote:
> > > > >>
> > > > >> > Hi All,
> > > > >> >
> > > > >> > I know what I'm about to ask may seem like crazy talk, but
I
> have
> > a
> > > > >> > situation at work where our CloudStack "box" was moved to
> another
> > > > >> network.
> > > > >> > This single server is a self-contained cloud with everything
on
> > one
> > > > box.
> > > > >> > I'm running CS 4.1.1 on CentOS 6.4 x86_64 with KVM and NFS
for
> > > > >> > primary
> > > > >> and
> > > > >> > secondary storage mounts. It's a very basic configuration,
but
> my
> > > > >> company
> > > > >> > moved from one town to another, and the IT dept decided
to move
> > our
> > > > >> server
> > > > >> > to a different switch and subnet without asking. So now
> literally
> > > > >> > every single IP address of any importance has been changed.
> > > > >> >
> > > > >> > I've been trying in vain to get this CS up and running again,
> but
> > > > >> > it
> > > > >> seems
> > > > >> > like everything has changed including the guids, or at least
> that
> > > > >> > seems
> > > > >> to
> > > > >> > be the case since I cannot match up guids from the database
to
> > > > >> > guids on
> > > > >> the
> > > > >> > primary/secondary NFS.
> > > > >> >
> > > > >> > I'm sure if each piece were separate and moved one piece
at a
> > time,
> > > > >> > it would be fairly straightforward, but since this box is
> > > > >> > self-contained
> > > > >> and
> > > > >> > everything changed at the same time, it is proving to be
very
> > > > difficult.
> > > > >> >
> > > > >> > Does anyone have any suggestions for me to attempt to recover
my
> > CS?
> > > > >> >
> > > > >> > Best Regards,
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Adam Scarcella
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Andrija Panić
> > > > >> --------------------------------------
> > > > >>   http://admintweets.com
> > > > >> --------------------------------------
> > > > >>
> > > > >
> > > > >
> > > > This email and any attachments to it may be confidential and are
> > intended
> > > > solely for the use of the individual to whom it is addressed. Any
> views
> > > or
> > > > opinions expressed are solely those of the author and do not
> > necessarily
> > > > represent those of Shape Blue Ltd or related companies. If you are
> not
> > > the
> > > > intended recipient of this email, you must neither take any action
> > based
> > > > upon its contents, nor copy or show it to anyone. Please contact the
> > > sender
> > > > if you believe you have received this email in error. Shape Blue Ltd
> > is a
> > > > company incorporated in England & Wales. ShapeBlue Services India
LLP
> > is
> > > a
> > > > company incorporated in India and is operated under license from
> Shape
> > > Blue
> > > > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
> > > Brasil
> > > > and is operated under license from Shape Blue Ltd. ShapeBlue is a
> > > > registered trademark.
> > > >
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> > --------------------------------------
> >   http://admintweets.com
> > --------------------------------------
> >
>



-- 

Andrija Panić
--------------------------------------
  http://admintweets.com
--------------------------------------

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message