cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alena Prokharchyk <Alena.Prokharc...@citrix.com>
Subject Re: [SOLVED] Unable to add an instance - speed=500Mhz, Ram=512 (host is in avoid set)
Date Tue, 05 Jun 2012 23:21:55 GMT
On 6/5/12 4:17 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:

>FYI.
>
>Alena manually removed the expunging instances from my CloudStack
>Management Server.


Small correction. I didn't manually removed the instances, I just
triggered expunge thread to run every 60 seconds, and the vms got expunged
by system following normal expunge process.

The one thing I did via mysql db - set up expunge.delay/expunge.inteval to
60 in cloud.config table; but it's also possible to do on Configuration
tab in cloudStack UI. I should have done it the official way rather then
mocking up with the DB.

-Alena.


> As a result, two allocated ip addresses were
>freed up. Then, I could add the VM instance without error (without
>an insufficient resources error). Essentially, the default time
>for the system to remove any expunging instances had not been
>reached yet. The manual, removal steps involved executing MySQL
>commands that updated a particular cloud table.
>
>This issue is resolved. Thanks, Alena.
>
>Regards,
>Evan
>
>-----Original Message-----
>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>Sent: Tuesday, June 05, 2012 1:14 PM
>To: cloudstack-users@incubator.apache.org
>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host is
>in avoid set)
>
>On 6/5/12 12:52 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>
>>Attached is the job-18 log snippet.
>>Sorry for not including this attachment with the previous email.
>>
>>Regards,
>>Evan
>>-----Original Message-----
>>From: Evan Miller [mailto:Evan.Miller@citrix.com]
>>Sent: Tuesday, June 05, 2012 12:48 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: RE: Unable to add an instance - speed=500Mhz, Ram=512 (host is
>>in avoid set)
>>
>>
>>-----Original Message-----
>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>Sent: Tuesday, June 05, 2012 12:08 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host is
>>in avoid set)
>>
>>On 6/5/12 11:49 AM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>
>>>RECOMMENDATION:
>>>Need to improve the 3.0.0 - 3.0.2 Basic Installation Guide.
>>>I don't have huge ranges of ip addresses available. So, it would be
>>>extremely beneficial to have some idea, in advance, as to how ip
>>>addresses will be consumed by a basic cloud setup. For example, I was
>>>surprised to learn that in my basic setup, one CloudStack Management
>>>Server (also serving primary and secondary filesystems) and one
>>>XenServer using one connected nic, the following ip addresses were
>>>required even before adding a single VM instance:
>>>
>>>  CSMS - 1 ip (management)  [not surprising]  XenServer - 1 ip
>>> (management) [not surprising]  s-1-VM - 3 ips (2 - management, 1 -
>>> guest) [surprising]  v-2-VM - 2 ips (1 - management, 1 - guest)
>>> [surprising]
>>>
>>>So, for each VM instance, is only one guest ip address required? Or is
>>>more required?
>>
>>Only one guest IP.
>>
>>>> After adding the guest IP to the pod, still not able to add a VM
>>>> instance to the cloud. Goth this popup error window:
>>
>>>>   Unable to create a deployment for VM[User|i-2-9-VM]
>>
>>>> Seeing insufficient capacity for job-18 in management-server.log
>>>> Attaching the log file snippet. It is hard for me to determine the
>>>> precise insufficiency. Please let me know.
>
>
>The insufficientCapacityException is the same - can't allocate Guest ip.
>Note then when the first user vm is being started in the Pod, and there
>is no Virtual Router running in the Pod, 2 ip addresses getting allocated:
>one for the Virtual router, another for user Vm.
>
>
>
>>
>>>> Also, the dashboard shows that the fourth guest ip address is still
>>>> taken. When will CSMS give back the ip address of the VM instance
>>>> that failed? Logging out and redisplaying the dashboard from the
>>>> browser still shows the fourth direct ip is taken. Destroying all
>>>> failed instances didn't help either. Clinging to that ip address
>>>> seems like a bug to me.
>
>
>The IP stays with the vm for the entire life cycle, and being released
>only when vm is Expunged. That's how we ensure ip address uniqueness for
>the vm for as long as vm lives. To make the ip release faster, change
>parameters for expunge thread (see below).
>
>
>
>
>
>>
>>>> Also, how long does expunging take after I destroy an instance?
>>>> That is, when does the instance finally become expunged?
>>>> I see instances expunging for about 24 hours now.
>
>
>
>Expunge thread runs every 24 hours; it's configurable. To make it faster,
>set expunge.delay/expunge.interval global config to some small values,
>say, 60 (seconds) and restart the management server process. The
>Destroyed vms should be expunged shortly.
>
>>
>>>> Regards,
>>>> Evan
>>
>>>
>>>Also, I, actually, don't see in XenCenter 6.0 what device is using the
>>>end (last) guest ip address that I defined. However, it is apparently
>>>used by something. What is using the last
>>>(end) guest ip address? The dashboard says the 3 direct ip addresses
>>>are used, but I can't identify what is using the third (last or end)
>>>guest ip address.
>>
>>
>>
>>You can check cloud.nics table (select instance_id where
>>ip4_address=<ip> and network_id=<id of the guest network>
>>
>>>
>>>
>>>Regards,
>>>Evan
>>>
>>>-----Original Message-----
>>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>>Sent: Monday, June 04, 2012 9:14 PM
>>>To: cloudstack-users@incubator.apache.org
>>>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host
>>>is in avoid set)
>>>
>>>On 6/4/12 8:07 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>>
>>>>Attached is the proper grep of job-14.
>>>>
>>>>In this small size instance scenario failure, is the XenServer the
>>>>system that is saying 'Hey 500Mbytes isn't enough RAM'?
>>>>
>>>>I am trying to understand exactly which system is saying what in
>>>>particular is insufficient.
>>>
>>>
>>>
>>>Here is the reason for the failure: Nic reservation failed due to
>>>Insufficient capacity exception (DirectPodBasedNetworkGuru.getIp:158):
>>>
>>>
>>>2012-06-04 18:10:02,577 DEBUG [db.Transaction.Transaction]
>>>(Job-Executor-1:job-14) Rolling back the transaction: Time = 2 Name =
>>>-AsyncJobManagerImpl$1.run:393-Executors$RunnableAdapter.call:471-Futu
>>>r
>>>eTa
>>>s
>>>k$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:11
>>>1
>>>0-T
>>>h
>>>readPoolExecutor$Worker.run:603-Thread.run:679; called by
>>>-Transaction.rollback:837-Transaction.removeUpTo:780-Transaction.close:
>>>599
>>>-
>>>DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-Ne
>>>t
>>>wor
>>>k
>>>ManagerImpl.assignPublicIpAddress:355-DirectPodBasedNetworkGuru.getIp:
>>>1
>>>58-
>>>D
>>>irectPodBasedNetworkGuru.reserve:146-DatabaseCallback.intercept:30-Net
>>>w
>>>ork
>>>M
>>>anagerImpl.prepare:1859-VirtualMachineManagerImpl.advanceStart:741-Vir
>>>t
>>>ual
>>>M
>>>achineManagerImpl.start:461
>>>2012-06-04 18:10:02,578 INFO  [cloud.vm.VirtualMachineManagerImpl]
>>>(Job-Executor-1:job-14) Insufficient capacity
>>>
>>>
>>>Looks like the Guest network doesn't have free guest IPs in the POD1
>>>targeted for deployment. To add the additional Ips, do this:
>>>
>>>* select Zone
>>>* Select Guest Network on the network diagram
>>>* Add guest IP range to the POD.
>>>
>>>
>>>-Alena.
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>Regards,
>>>>Evan
>>>>
>>>>-----Original Message-----
>>>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>>>Sent: Monday, June 04, 2012 7:41 PM
>>>>To: cloudstack-users@incubator.apache.org
>>>>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host
>>>>is in avoid set)
>>>>
>>>>The setup lacks the resources. See more detailed information inline.
>>>>Btw, to capture all logs related to vm deployment, grep by job-id
>>>>should be done (in this case "grep job-14 management-server.log").
>>>>
>>>>-Alena.
>>>>
>>>>
>>>>On 6/4/12 7:27 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>>>
>>>>>Trying to setup a single, small VM to my cloud.
>>>>>My cloud, so far, consists of:
>>>>>
>>>>>CloudStack Management Server:
>>>>> *   OS = Centos 6.2 x86 64-bit
>>>>> *   CPU = Intel 3050, 2.13 GHz
>>>>> *   RAM = 2 Gbytes
>>>>>
>>>>>XenServer:
>>>>> *   OS = Provided by XenServer 6.0.2 CD
>>>>> *   Hardware = IBM 3250
>>>>> *   CPU = Intel Xeon Dual Core, Single Physical CPU@ 1.86 GHz
>>>>>
>>>>>Template:
>>>>>* CentOS 5.6(64-bit) no GUI (XenServer)
>>>>>
>>>>>Per the management log below, is XenServer refusing to setup the
>>>>>first VM?
>>>>>
>>>>>
>>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) DeploymentPlanner allocation algorithm:
>>>>>random
>>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) Trying to allocate a host and storage pools
>>>>>from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram:
>>>>>536870912
>>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) Is ROOT volume READY (pool already
>>>>>allocated)?:
>>>>>No
>>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) Searching all possible resources under this
>>>>>Zone:
>>>>>1
>>>>>2012-06-04 18:10:02,909 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) Listing clusters in order of aggregate
>>>>>capacity, that have (atleast one host with) enough CPU and RAM
>>>>>capacity under this
>>>>>Zone: 1
>>>>>2012-06-04 18:10:02,910 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) CPUOverprovisioningFactor considered: 1.0
>>>>>2012-06-04 18:10:02,918 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) Checking resources in Cluster: 1 under Pod:
>>>>>1
>>>>>2012-06-04 18:10:02,918 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) Calling HostAllocators to find suitable
>>>>>hosts
>>>>>2012-06-04 18:10:02,918 DEBUG [allocator.impl.FirstFitAllocator]
>>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Looking for hosts
>>>>>in
>>>>>dc:
>>>>>1  pod:1  cluster:1
>>>>>2012-06-04 18:10:02,920 DEBUG [allocator.impl.FirstFitAllocator]
>>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) FirstFitAllocator
>>>>>has
>>>>>1 hosts to check for allocation: [Host[-1-Routing]]
>>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Found 1 hosts for
>>>>>allocation after prioritization: [Host[-1-Routing]]
>>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Looking for
>>>>>speed=500Mhz, Ram=512
>>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Host name:
>>>>>xenserver1-cs, hostId: 1 is in avoid set, skipping this and trying
>>>>>other available hosts
>>>>
>>>>
>>>>The host was put in avoid set some time before (the log fragment
>>>>you've sent doesn't capture it). The host can be put in avoid set
>>>>when it doesn't have enough resources for vm start (cpu/ram).
>>>>
>>>>
>>>>
>>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Host Allocator
>>>>>returning 0 suitable hosts
>>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) No suitable hosts found
>>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) No suitable hosts found under this Cluster:
>>>>>1
>>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-1:job-14) Could not find suitable Deployment
>>>>>Destination for this VM under any clusters, returning.
>>>>>2012-06-04 18:10:03,022 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-1:job-14) VM state transitted from :Starting to
>>>>>Stopped with event: OperationFailedvm's original host id: null new
>>>>>host id:
>>>>>null host id before state transition: 1
>>>>>2012-06-04 18:10:03,027 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-1:job-14) Hosts's actual total CPU: 3732 and CPU after
>>>>>applying overprovisioning: 3732
>>>>>2012-06-04 18:10:03,028 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-1:job-14) release cpu from host: 1, old used:
>>>>>1500,reserved: 0, actual total: 3732, total with overprovisioning:
>>>>>3732; new used: 1000,reserved:0; movedfromreserved:
>>>>>false,moveToReserveredfalse
>>>>>2012-06-04 18:10:03,028 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-1:job-14) release mem from host: 1, old used:
>>>>>1879048192,reserved: 0, total: 3775173696; new used:
>>>>>1342177280,reserved:0; movedfromreserved:
>>>>>false,moveToReserveredfalse
>>>>>2012-06-04 18:10:03,183 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-1:job-14) VM state transitted from :Stopped to Error
>>>>>with
>>>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>>>null host id before state transition: null
>>>>>2012-06-04 18:10:03,888 INFO  [api.commands.DeployVMCmd]
>>>>>(Job-Executor-1:job-14)
>>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to
>>>>>create a deployment for VM[User|i-2-7-VM]Scope=interface
>>>>>com.cloud.dc.DataCenter; id=1
>>>>>2012-06-04 18:10:03,888 WARN  [cloud.api.ApiDispatcher]
>>>>>(Job-Executor-1:job-14) class com.cloud.api.ServerApiException :
>>>>>Unable to create a deployment for VM[User|i-2-7-VM]
>>>>>2012-06-04 18:10:03,891 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>>(Job-Executor-1:job-14) Complete async job-14, jobStatus: 2,
>>>>>resultCode:
>>>>>530, result:
>>>>>com.cloud.api.response.ExceptionResponse@4ebe0382<mailto:com.cloud.api
>>>>>.
>>>>>res
>>>>>ponse.ExceptionResponse@4ebe0382>
>>>>>2012-06-04 18:10:07,258 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>>(catalina-exec-7:null) Async job-14 completed
>>>>>
>>>>>
>>>>>
>>>>>I, also, tried to create a medium size VM. This, also, failed with a
>>>>>different error related to InsufficientServerCapacityException:
>>>>>
>>>>>
>>>>>
>>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) DeploymentPlanner allocation algorithm:
>>>>>random
>>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) Trying to allocate a host and storage pools
>>>>>from dc:1, pod:null,cluster:null, requested cpu: 1000, requested ram:
>>>>>1073741824
>>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) Is ROOT volume READY (pool already allocated)?:
>>>>>No
>>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) Searching all possible resources under this
>>>>>Zone: 1
>>>>>2012-06-04 17:34:52,551 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) Listing clusters in order of aggregate
>>>>>capacity, that have (atleast one host with) enough CPU and RAM
>>>>>capacity under this
>>>>>Zone: 1
>>>>>2012-06-04 17:34:52,552 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) CPUOverprovisioningFactor considered: 1.0
>>>>>2012-06-04 17:34:52,558 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) Cannot allocate cluster 1 for vm creation
>>>>>since its allocated percentage: 0.9243709585329766 will cross the
>>>>>disable capacity threshold: 0.85 for capacity Type : 0, skipping
>>>>>this cluster
>>>>
>>>>
>>>>CapacityType: 0 means RAM. When we analyze if host is capable for vm
>>>>deployment, we always make sure the total RAM (including the RAM we
>>>>are allocating for the vm), doesn't exceed RAM capacity threshold
>>>>(global config variable
>>>>cluster.memory.allocated.capacity.disablethreshold).
>>>>
>>>>
>>>>
>>>>>2012-06-04 17:34:52,558 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>>(Job-Executor-9:job-9) Could not find suitable Deployment
>>>>>Destination for this VM under any clusters, returning.
>>>>>2012-06-04 17:34:52,690 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-9:job-9) VM state transitted from :Starting to Stopped
>>>>>with
>>>>>event: OperationFailedvm's original host id: null new host id: null
>>>>>host id before state transition: 1
>>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-9:job-9) Hosts's actual total CPU: 3732 and CPU after
>>>>>applying overprovisioning: 3732
>>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-9:job-9) release cpu from host: 1, old used:
>>>>>2000,reserved:
>>>>>0, actual total: 3732, total with overprovisioning: 3732; new used:
>>>>>1000,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-9:job-9) release mem from host: 1, old used:
>>>>>2415919104,reserved: 0, total: 3775173696; new used:
>>>>>1342177280,reserved:0; movedfromreserved:
>>>>>false,moveToReserveredfalse
>>>>>2012-06-04 17:34:52,842 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-9:job-9) VM state transitted from :Stopped to Error
>>>>>with
>>>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>>>null host id before state transition: null
>>>>>2012-06-04 17:34:53,463 INFO  [api.commands.DeployVMCmd]
>>>>>(Job-Executor-9:job-9)
>>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to
>>>>>create a deployment for VM[User|i-2-5-VM]Scope=interface
>>>>>com.cloud.dc.DataCenter; id=1
>>>>>2012-06-04 17:34:53,464 WARN  [cloud.api.ApiDispatcher]
>>>>>(Job-Executor-9:job-9) class com.cloud.api.ServerApiException :
>>>>>Unable to create a deployment for VM[User|i-2-5-VM]
>>>>>2012-06-04 17:34:53,464 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>>(Job-Executor-9:job-9) Complete async job-9, jobStatus: 2, resultCode:
>>>>>530, result:
>>>>>com.cloud.api.response.ExceptionResponse@7f378496<mailto:com.cloud.api
>>>>>.
>>>>>res
>>>>>ponse.ExceptionResponse@7f378496>
>>>>>2012-06-04 17:34:56,985 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>>(catalina-exec-17:null) Async job-9 completed
>>>>>
>>>>>How to overcome these errors?
>>>>
>>>>
>>>>The solution will be - create the service offering with small amount
>>>>of RAM/CPU (depends of how many vms you are planning to run on your
>>>>host).
>>>>Or add more hosts to the Xen cluster.
>>>>
>>>>>
>>>>>Regards,
>>>>>
>>>>>Evan Miller
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>



Mime
View raw message