cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Miller <Evan.Mil...@citrix.com>
Subject RE: [SOLVED] Unable to add an instance - speed=500Mhz, Ram=512 (host is in avoid set)
Date Tue, 05 Jun 2012 23:17:16 GMT
FYI.

Alena manually removed the expunging instances from my CloudStack
Management Server. As a result, two allocated ip addresses were
freed up. Then, I could add the VM instance without error (without
an insufficient resources error). Essentially, the default time
for the system to remove any expunging instances had not been
reached yet. The manual, removal steps involved executing MySQL
commands that updated a particular cloud table.

This issue is resolved. Thanks, Alena.

Regards,
Evan

-----Original Message-----
From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
Sent: Tuesday, June 05, 2012 1:14 PM
To: cloudstack-users@incubator.apache.org
Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host is in avoid set)

On 6/5/12 12:52 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:

>Attached is the job-18 log snippet.
>Sorry for not including this attachment with the previous email.
>
>Regards,
>Evan
>-----Original Message-----
>From: Evan Miller [mailto:Evan.Miller@citrix.com]
>Sent: Tuesday, June 05, 2012 12:48 PM
>To: cloudstack-users@incubator.apache.org
>Subject: RE: Unable to add an instance - speed=500Mhz, Ram=512 (host is
>in avoid set)
>
>
>-----Original Message-----
>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>Sent: Tuesday, June 05, 2012 12:08 PM
>To: cloudstack-users@incubator.apache.org
>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host is
>in avoid set)
>
>On 6/5/12 11:49 AM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>
>>RECOMMENDATION:
>>Need to improve the 3.0.0 - 3.0.2 Basic Installation Guide.
>>I don't have huge ranges of ip addresses available. So, it would be
>>extremely beneficial to have some idea, in advance, as to how ip
>>addresses will be consumed by a basic cloud setup. For example, I was
>>surprised to learn that in my basic setup, one CloudStack Management
>>Server (also serving primary and secondary filesystems) and one
>>XenServer using one connected nic, the following ip addresses were
>>required even before adding a single VM instance:
>>
>>  CSMS - 1 ip (management)  [not surprising]  XenServer - 1 ip
>> (management) [not surprising]  s-1-VM - 3 ips (2 - management, 1 -
>> guest) [surprising]  v-2-VM - 2 ips (1 - management, 1 - guest)
>> [surprising]
>>
>>So, for each VM instance, is only one guest ip address required? Or is
>>more required?
>
>Only one guest IP.
>
>>> After adding the guest IP to the pod, still not able to add a VM
>>> instance to the cloud. Goth this popup error window:
>
>>>   Unable to create a deployment for VM[User|i-2-9-VM]
>
>>> Seeing insufficient capacity for job-18 in management-server.log
>>> Attaching the log file snippet. It is hard for me to determine the
>>> precise insufficiency. Please let me know.


The insufficientCapacityException is the same - can't allocate Guest ip.
Note then when the first user vm is being started in the Pod, and there is no Virtual Router
running in the Pod, 2 ip addresses getting allocated:
one for the Virtual router, another for user Vm.



>
>>> Also, the dashboard shows that the fourth guest ip address is still
>>> taken. When will CSMS give back the ip address of the VM instance
>>> that failed? Logging out and redisplaying the dashboard from the
>>> browser still shows the fourth direct ip is taken. Destroying all
>>> failed instances didn't help either. Clinging to that ip address
>>> seems like a bug to me.


The IP stays with the vm for the entire life cycle, and being released only when vm is Expunged.
That's how we ensure ip address uniqueness for the vm for as long as vm lives. To make the
ip release faster, change parameters for expunge thread (see below).





>
>>> Also, how long does expunging take after I destroy an instance?
>>> That is, when does the instance finally become expunged?
>>> I see instances expunging for about 24 hours now.



Expunge thread runs every 24 hours; it's configurable. To make it faster, set expunge.delay/expunge.interval
global config to some small values, say, 60 (seconds) and restart the management server process.
The Destroyed vms should be expunged shortly.

>
>>> Regards,
>>> Evan
>
>>
>>Also, I, actually, don't see in XenCenter 6.0 what device is using the
>>end (last) guest ip address that I defined. However, it is apparently
>>used by something. What is using the last
>>(end) guest ip address? The dashboard says the 3 direct ip addresses
>>are used, but I can't identify what is using the third (last or end)
>>guest ip address.
>
>
>
>You can check cloud.nics table (select instance_id where
>ip4_address=<ip> and network_id=<id of the guest network>
>
>>
>>
>>Regards,
>>Evan
>>
>>-----Original Message-----
>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>Sent: Monday, June 04, 2012 9:14 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host
>>is in avoid set)
>>
>>On 6/4/12 8:07 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>
>>>Attached is the proper grep of job-14.
>>>
>>>In this small size instance scenario failure, is the XenServer the
>>>system that is saying 'Hey 500Mbytes isn't enough RAM'?
>>>
>>>I am trying to understand exactly which system is saying what in
>>>particular is insufficient.
>>
>>
>>
>>Here is the reason for the failure: Nic reservation failed due to
>>Insufficient capacity exception (DirectPodBasedNetworkGuru.getIp:158):
>>
>>
>>2012-06-04 18:10:02,577 DEBUG [db.Transaction.Transaction]
>>(Job-Executor-1:job-14) Rolling back the transaction: Time = 2 Name =
>>-AsyncJobManagerImpl$1.run:393-Executors$RunnableAdapter.call:471-Futu
>>r
>>eTa
>>s
>>k$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:11
>>1
>>0-T
>>h
>>readPoolExecutor$Worker.run:603-Thread.run:679; called by
>>-Transaction.rollback:837-Transaction.removeUpTo:780-Transaction.close:
>>599
>>-
>>DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-Ne
>>t
>>wor
>>k
>>ManagerImpl.assignPublicIpAddress:355-DirectPodBasedNetworkGuru.getIp:
>>1
>>58-
>>D
>>irectPodBasedNetworkGuru.reserve:146-DatabaseCallback.intercept:30-Net
>>w
>>ork
>>M
>>anagerImpl.prepare:1859-VirtualMachineManagerImpl.advanceStart:741-Vir
>>t
>>ual
>>M
>>achineManagerImpl.start:461
>>2012-06-04 18:10:02,578 INFO  [cloud.vm.VirtualMachineManagerImpl]
>>(Job-Executor-1:job-14) Insufficient capacity
>>
>>
>>Looks like the Guest network doesn't have free guest IPs in the POD1
>>targeted for deployment. To add the additional Ips, do this:
>>
>>* select Zone
>>* Select Guest Network on the network diagram
>>* Add guest IP range to the POD.
>>
>>
>>-Alena.
>>
>>
>>
>>
>>>
>>>
>>>Regards,
>>>Evan
>>>
>>>-----Original Message-----
>>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>>Sent: Monday, June 04, 2012 7:41 PM
>>>To: cloudstack-users@incubator.apache.org
>>>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host
>>>is in avoid set)
>>>
>>>The setup lacks the resources. See more detailed information inline.
>>>Btw, to capture all logs related to vm deployment, grep by job-id
>>>should be done (in this case "grep job-14 management-server.log").
>>>
>>>-Alena.
>>>
>>>
>>>On 6/4/12 7:27 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>>
>>>>Trying to setup a single, small VM to my cloud.
>>>>My cloud, so far, consists of:
>>>>
>>>>CloudStack Management Server:
>>>> *   OS = Centos 6.2 x86 64-bit
>>>> *   CPU = Intel 3050, 2.13 GHz
>>>> *   RAM = 2 Gbytes
>>>>
>>>>XenServer:
>>>> *   OS = Provided by XenServer 6.0.2 CD
>>>> *   Hardware = IBM 3250
>>>> *   CPU = Intel Xeon Dual Core, Single Physical CPU@ 1.86 GHz
>>>>
>>>>Template:
>>>>* CentOS 5.6(64-bit) no GUI (XenServer)
>>>>
>>>>Per the management log below, is XenServer refusing to setup the
>>>>first VM?
>>>>
>>>>
>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) DeploymentPlanner allocation algorithm:
>>>>random
>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) Trying to allocate a host and storage pools
>>>>from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram:
>>>>536870912
>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) Is ROOT volume READY (pool already allocated)?:
>>>>No
>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) Searching all possible resources under this
>>>>Zone:
>>>>1
>>>>2012-06-04 18:10:02,909 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) Listing clusters in order of aggregate
>>>>capacity, that have (atleast one host with) enough CPU and RAM
>>>>capacity under this
>>>>Zone: 1
>>>>2012-06-04 18:10:02,910 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) CPUOverprovisioningFactor considered: 1.0
>>>>2012-06-04 18:10:02,918 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) Checking resources in Cluster: 1 under Pod:
>>>>1
>>>>2012-06-04 18:10:02,918 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) Calling HostAllocators to find suitable
>>>>hosts
>>>>2012-06-04 18:10:02,918 DEBUG [allocator.impl.FirstFitAllocator]
>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Looking for hosts
>>>>in
>>>>dc:
>>>>1  pod:1  cluster:1
>>>>2012-06-04 18:10:02,920 DEBUG [allocator.impl.FirstFitAllocator]
>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) FirstFitAllocator
>>>>has
>>>>1 hosts to check for allocation: [Host[-1-Routing]]
>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Found 1 hosts for
>>>>allocation after prioritization: [Host[-1-Routing]]
>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Looking for
>>>>speed=500Mhz, Ram=512
>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Host name:
>>>>xenserver1-cs, hostId: 1 is in avoid set, skipping this and trying
>>>>other available hosts
>>>
>>>
>>>The host was put in avoid set some time before (the log fragment
>>>you've sent doesn't capture it). The host can be put in avoid set
>>>when it doesn't have enough resources for vm start (cpu/ram).
>>>
>>>
>>>
>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator]
>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Host Allocator
>>>>returning 0 suitable hosts
>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) No suitable hosts found
>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) No suitable hosts found under this Cluster:
>>>>1
>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-1:job-14) Could not find suitable Deployment
>>>>Destination for this VM under any clusters, returning.
>>>>2012-06-04 18:10:03,022 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-1:job-14) VM state transitted from :Starting to
>>>>Stopped with event: OperationFailedvm's original host id: null new host id:
>>>>null host id before state transition: 1
>>>>2012-06-04 18:10:03,027 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-1:job-14) Hosts's actual total CPU: 3732 and CPU after
>>>>applying overprovisioning: 3732
>>>>2012-06-04 18:10:03,028 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-1:job-14) release cpu from host: 1, old used:
>>>>1500,reserved: 0, actual total: 3732, total with overprovisioning:
>>>>3732; new used: 1000,reserved:0; movedfromreserved:
>>>>false,moveToReserveredfalse
>>>>2012-06-04 18:10:03,028 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-1:job-14) release mem from host: 1, old used:
>>>>1879048192,reserved: 0, total: 3775173696; new used:
>>>>1342177280,reserved:0; movedfromreserved:
>>>>false,moveToReserveredfalse
>>>>2012-06-04 18:10:03,183 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-1:job-14) VM state transitted from :Stopped to Error
>>>>with
>>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>>null host id before state transition: null
>>>>2012-06-04 18:10:03,888 INFO  [api.commands.DeployVMCmd]
>>>>(Job-Executor-1:job-14)
>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to
>>>>create a deployment for VM[User|i-2-7-VM]Scope=interface
>>>>com.cloud.dc.DataCenter; id=1
>>>>2012-06-04 18:10:03,888 WARN  [cloud.api.ApiDispatcher]
>>>>(Job-Executor-1:job-14) class com.cloud.api.ServerApiException :
>>>>Unable to create a deployment for VM[User|i-2-7-VM]
>>>>2012-06-04 18:10:03,891 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>(Job-Executor-1:job-14) Complete async job-14, jobStatus: 2,
>>>>resultCode:
>>>>530, result:
>>>>com.cloud.api.response.ExceptionResponse@4ebe0382<mailto:com.cloud.api.
>>>>res
>>>>ponse.ExceptionResponse@4ebe0382>
>>>>2012-06-04 18:10:07,258 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>(catalina-exec-7:null) Async job-14 completed
>>>>
>>>>
>>>>
>>>>I, also, tried to create a medium size VM. This, also, failed with a
>>>>different error related to InsufficientServerCapacityException:
>>>>
>>>>
>>>>
>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) DeploymentPlanner allocation algorithm:
>>>>random
>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) Trying to allocate a host and storage pools
>>>>from dc:1, pod:null,cluster:null, requested cpu: 1000, requested ram:
>>>>1073741824
>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) Is ROOT volume READY (pool already allocated)?:
>>>>No
>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) Searching all possible resources under this
>>>>Zone: 1
>>>>2012-06-04 17:34:52,551 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) Listing clusters in order of aggregate
>>>>capacity, that have (atleast one host with) enough CPU and RAM
>>>>capacity under this
>>>>Zone: 1
>>>>2012-06-04 17:34:52,552 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) CPUOverprovisioningFactor considered: 1.0
>>>>2012-06-04 17:34:52,558 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) Cannot allocate cluster 1 for vm creation
>>>>since its allocated percentage: 0.9243709585329766 will cross the
>>>>disable capacity threshold: 0.85 for capacity Type : 0, skipping
>>>>this cluster
>>>
>>>
>>>CapacityType: 0 means RAM. When we analyze if host is capable for vm
>>>deployment, we always make sure the total RAM (including the RAM we
>>>are allocating for the vm), doesn't exceed RAM capacity threshold
>>>(global config variable
>>>cluster.memory.allocated.capacity.disablethreshold).
>>>
>>>
>>>
>>>>2012-06-04 17:34:52,558 DEBUG [cloud.deploy.FirstFitPlanner]
>>>>(Job-Executor-9:job-9) Could not find suitable Deployment
>>>>Destination for this VM under any clusters, returning.
>>>>2012-06-04 17:34:52,690 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-9:job-9) VM state transitted from :Starting to Stopped
>>>>with
>>>>event: OperationFailedvm's original host id: null new host id: null
>>>>host id before state transition: 1
>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-9:job-9) Hosts's actual total CPU: 3732 and CPU after
>>>>applying overprovisioning: 3732
>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-9:job-9) release cpu from host: 1, old used:
>>>>2000,reserved:
>>>>0, actual total: 3732, total with overprovisioning: 3732; new used:
>>>>1000,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-9:job-9) release mem from host: 1, old used:
>>>>2415919104,reserved: 0, total: 3775173696; new used:
>>>>1342177280,reserved:0; movedfromreserved:
>>>>false,moveToReserveredfalse
>>>>2012-06-04 17:34:52,842 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-9:job-9) VM state transitted from :Stopped to Error
>>>>with
>>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>>null host id before state transition: null
>>>>2012-06-04 17:34:53,463 INFO  [api.commands.DeployVMCmd]
>>>>(Job-Executor-9:job-9)
>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to
>>>>create a deployment for VM[User|i-2-5-VM]Scope=interface
>>>>com.cloud.dc.DataCenter; id=1
>>>>2012-06-04 17:34:53,464 WARN  [cloud.api.ApiDispatcher]
>>>>(Job-Executor-9:job-9) class com.cloud.api.ServerApiException :
>>>>Unable to create a deployment for VM[User|i-2-5-VM]
>>>>2012-06-04 17:34:53,464 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>(Job-Executor-9:job-9) Complete async job-9, jobStatus: 2, resultCode:
>>>>530, result:
>>>>com.cloud.api.response.ExceptionResponse@7f378496<mailto:com.cloud.api.
>>>>res
>>>>ponse.ExceptionResponse@7f378496>
>>>>2012-06-04 17:34:56,985 DEBUG [cloud.async.AsyncJobManagerImpl]
>>>>(catalina-exec-17:null) Async job-9 completed
>>>>
>>>>How to overcome these errors?
>>>
>>>
>>>The solution will be - create the service offering with small amount
>>>of RAM/CPU (depends of how many vms you are planning to run on your
>>>host).
>>>Or add more hosts to the Xen cluster.
>>>
>>>>
>>>>Regards,
>>>>
>>>>Evan Miller
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>



Mime
View raw message